Skip to content

Configure Auto-Compaction

Auto-compaction summarizes older turns when a long conversation approaches the active model's context limit. It is disabled by default (COMPACT_ENABLED=false). When it runs, it stores a non-destructive checkpoint summary in SQLite while keeping all original messages in the database.

Pack a summary and recent messages into a limited context while keeping the originals.

Enable compaction

Set COMPACT_ENABLED=true in .env and restart Numi Chat:

COMPACT_ENABLED=true

Whenever a model entry in the model catalog specifies its own context_window, Numi uses that value; otherwise it falls back to the per-provider window below. (The legacy DISABLE_AUTO_COMPACT=1 environment variable also disables auto-compaction if set.)

How the threshold and token estimate work

window    = catalog context_window, otherwise provider fallback
threshold = max(1, window - COMPACT_BUFFER_TOKENS)

Before each model step, the agent estimates the size of the prepared request (messages + active tool schemas) and triggers compaction when the estimate exceeds threshold:

  • Quad-token estimation (count_quad_tokens): Every 4 printable ASCII characters count as 1 token, every non-ASCII code point counts as 1 token on its own, dictionary keys are counted, and each image part (image_url / input_image) is budgeted at 4,096 tokens.
  • History character scaling (_history_char_budget): When loading history from SQLite, the character cap scales automatically to max(PROMPT_MAX_HISTORY_CHARS, threshold * 3) so models with large context windows are not truncated prematurely before reaching their compaction threshold.
  • Fallback when compaction is off: Even when auto-compaction is disabled or fails, the request builder drops older complete turns if needed to stay within the history budget.

Per-provider fallback context windows

When a model's catalog entry has context_window: null, Numi uses the provider fallback:

Provider Environment variable Default (tokens)
Crowbot COMPACT_CONTEXT_WINDOW_CROWBOT 1000000
Anthropic COMPACT_CONTEXT_WINDOW_ANTHROPIC 1000000
Anthropic-compatible COMPACT_CONTEXT_WINDOW_ANTHROPIC_COMPAT 1000000
OpenRouter COMPACT_CONTEXT_WINDOW_OPENROUTER 256000
Z.AI COMPACT_CONTEXT_WINDOW_ZAI 128000
OpenAI COMPACT_CONTEXT_WINDOW_OPENAI 128000
Responses COMPACT_CONTEXT_WINDOW_RESPONSES 128000
Mistral COMPACT_CONTEXT_WINDOW_MISTRAL 128000
Inception COMPACT_CONTEXT_WINDOW_INCEPTION 32000
Local COMPACT_CONTEXT_WINDOW_LOCAL 32000

Because a single provider (like OpenRouter or Local) can serve models with very different window sizes, prefer setting context_window on each model entry in the catalog rather than raising a provider-wide fallback.

Tuning COMPACT_BUFFER_TOKENS and COMPACT_KEEP_RECENT

COMPACT_BUFFER_TOKENS (default 13000)

Reserves headroom below the context window for the model's output, reasoning tokens, and estimation margin:

  • Decrease the buffer if compaction triggers too early.
  • Increase the buffer if requests hit upstream context-length errors before compacting.

COMPACT_KEEP_RECENT (default 12)

Sets the minimum number of recent non-system messages kept verbatim after the summary. Numi walks the split point back to a user message boundary so complete turns (including their tool_calls and tool results) are never cut in half.

Compaction requires at least COMPACT_KEEP_RECENT + 4 non-system messages and at least one older complete turn to summarize. After compaction, the active prompt history consists of:

1. System message
2. Structured summary of older turns
3. Assistant acknowledgement
4. Recent complete turns (at least COMPACT_KEEP_RECENT messages)

Example configurations

Small local model (4K context)

COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_LOCAL=4096
COMPACT_BUFFER_TOKENS=1024
COMPACT_KEEP_RECENT=6

Compaction triggers at 3,072 estimated tokens.

Custom OpenRouter threshold (128K context)

COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_OPENROUTER=128000
COMPACT_BUFFER_TOKENS=16000
COMPACT_KEEP_RECENT=12

Compaction triggers at 112,000 estimated tokens.

Circuit breaker and persistence

If a summarization call fails or returns an empty response, Numi increments a failure counter (COMPACT_MAX_FAILURES, default 3) and records a 5-minute retry cooldown on the chat's compaction checkpoint. Once COMPACT_MAX_FAILURES is reached, the agent skips compaction for the remainder of the turn and continues with normal turn-dropping bounds.

When summarization succeeds, the 9-section <summary> and the highest covered message.id (through_message_id) are persisted to the chatcompaction table (src/numi_chat/chat/context.py), so subsequent turns reuse the summary without re-summarizing those messages.