Configure Auto-Compaction¶
Auto-compaction summarizes older turns when a long conversation approaches the
active model's context limit. It is disabled by default
(COMPACT_ENABLED=false). When it runs, it stores a non-destructive checkpoint
summary in SQLite while keeping all original messages in the database.
Enable compaction¶
Set COMPACT_ENABLED=true in .env and restart Numi Chat:
Whenever a model entry in the model catalog specifies its own context_window, Numi uses that value; otherwise it falls back to the per-provider window below. (The legacy DISABLE_AUTO_COMPACT=1 environment variable also disables auto-compaction if set.)
How the threshold and token estimate work¶
window = catalog context_window, otherwise provider fallback
threshold = max(1, window - COMPACT_BUFFER_TOKENS)
Before each model step, the agent estimates the size of the prepared request
(messages + active tool schemas) and triggers compaction when the estimate
exceeds threshold:
- Quad-token estimation (
count_quad_tokens): Every 4 printable ASCII characters count as 1 token, every non-ASCII code point counts as 1 token on its own, dictionary keys are counted, and each image part (image_url/input_image) is budgeted at 4,096 tokens. - History character scaling (
_history_char_budget): When loading history from SQLite, the character cap scales automatically tomax(PROMPT_MAX_HISTORY_CHARS, threshold * 3)so models with large context windows are not truncated prematurely before reaching their compaction threshold. - Fallback when compaction is off: Even when auto-compaction is disabled or fails, the request builder drops older complete turns if needed to stay within the history budget.
Per-provider fallback context windows¶
When a model's catalog entry has context_window: null, Numi uses the provider
fallback:
| Provider | Environment variable | Default (tokens) |
|---|---|---|
| Crowbot | COMPACT_CONTEXT_WINDOW_CROWBOT |
1000000 |
| Anthropic | COMPACT_CONTEXT_WINDOW_ANTHROPIC |
1000000 |
| Anthropic-compatible | COMPACT_CONTEXT_WINDOW_ANTHROPIC_COMPAT |
1000000 |
| OpenRouter | COMPACT_CONTEXT_WINDOW_OPENROUTER |
256000 |
| Z.AI | COMPACT_CONTEXT_WINDOW_ZAI |
128000 |
| OpenAI | COMPACT_CONTEXT_WINDOW_OPENAI |
128000 |
| Responses | COMPACT_CONTEXT_WINDOW_RESPONSES |
128000 |
| Mistral | COMPACT_CONTEXT_WINDOW_MISTRAL |
128000 |
| Inception | COMPACT_CONTEXT_WINDOW_INCEPTION |
32000 |
| Local | COMPACT_CONTEXT_WINDOW_LOCAL |
32000 |
Because a single provider (like OpenRouter or Local) can serve models with very
different window sizes, prefer setting context_window on each model entry in
the catalog rather than raising a provider-wide fallback.
Tuning COMPACT_BUFFER_TOKENS and COMPACT_KEEP_RECENT¶
COMPACT_BUFFER_TOKENS (default 13000)¶
Reserves headroom below the context window for the model's output, reasoning tokens, and estimation margin:
- Decrease the buffer if compaction triggers too early.
- Increase the buffer if requests hit upstream context-length errors before compacting.
COMPACT_KEEP_RECENT (default 12)¶
Sets the minimum number of recent non-system messages kept verbatim after the
summary. Numi walks the split point back to a user message boundary so
complete turns (including their tool_calls and tool results) are never cut
in half.
Compaction requires at least COMPACT_KEEP_RECENT + 4 non-system messages and
at least one older complete turn to summarize. After compaction, the active
prompt history consists of:
1. System message
2. Structured summary of older turns
3. Assistant acknowledgement
4. Recent complete turns (at least COMPACT_KEEP_RECENT messages)
Example configurations¶
Small local model (4K context)¶
COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_LOCAL=4096
COMPACT_BUFFER_TOKENS=1024
COMPACT_KEEP_RECENT=6
Compaction triggers at 3,072 estimated tokens.
Custom OpenRouter threshold (128K context)¶
COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_OPENROUTER=128000
COMPACT_BUFFER_TOKENS=16000
COMPACT_KEEP_RECENT=12
Compaction triggers at 112,000 estimated tokens.
Circuit breaker and persistence¶
If a summarization call fails or returns an empty response, Numi increments a
failure counter (COMPACT_MAX_FAILURES, default 3) and records a 5-minute
retry cooldown on the chat's compaction checkpoint. Once COMPACT_MAX_FAILURES
is reached, the agent skips compaction for the remainder of the turn and
continues with normal turn-dropping bounds.
When summarization succeeds, the 9-section <summary> and the highest covered
message.id (through_message_id) are persisted to the chatcompaction table
(src/numi_chat/chat/context.py), so subsequent turns reuse the summary without
re-summarizing those messages.