Skip to content

Configure Auto-Compaction

Auto-compaction summarizes older turns before a request approaches its context budget. It is disabled by default. Summarization can omit details; the original messages remain in the database.

Pack a summary and recent messages into a limited context while keeping the originals.

Enable compaction

Add this setting to your server environment, then restart Numi Chat:

COMPACT_ENABLED=true

Set the model's context_window in the catalog JSON when known, then import the catalog or reload the edited file. The Models form does not expose this field. The catalog value takes precedence over the per-provider fallback below. Choose a window supported by the endpoint actually serving the model.

The older DISABLE_AUTO_COMPACT process-environment switch still overrides this setting when set to 1, true, or yes.

How the threshold is calculated

window = catalog context_window, otherwise provider fallback
threshold = max(1, window - COMPACT_BUFFER_TOKENS)

The agent attempts compaction when its estimated request size is greater than this threshold. The estimate includes prepared messages and active tool schemas. It uses roughly one token per three characters, counts dictionary keys, and assigns 4,096 tokens to each image part. This is a budgeting heuristic, not the provider's tokenizer or an exact multimodal token count.

Context limits apply even when summarization is disabled or fails. The request builder drops older complete turns until a request fits. If the latest turn and tool schemas alone exceed the budget, it stops with an error asking for fewer attachments or a larger-context model.

Per-provider context windows

These are fallback values, not detected model limits:

Provider Environment variable Default
Z.AI COMPACT_CONTEXT_WINDOW_ZAI 128000
OpenRouter COMPACT_CONTEXT_WINDOW_OPENROUTER 128000
OpenAI COMPACT_CONTEXT_WINDOW_OPENAI 128000
Responses COMPACT_CONTEXT_WINDOW_RESPONSES 128000
Anthropic COMPACT_CONTEXT_WINDOW_ANTHROPIC 1000000
Anthropic-compatible COMPACT_CONTEXT_WINDOW_ANTHROPIC_COMPAT 1000000
Inception COMPACT_CONTEXT_WINDOW_INCEPTION 32000
Local COMPACT_CONTEXT_WINDOW_LOCAL 32000

A provider can serve several models with different windows. Prefer a catalog value per model instead of increasing one shared fallback for all of them.

Choosing COMPACT_BUFFER_TOKENS

The default buffer is 13000 tokens. It reserves room below the configured window for output and estimation error.

  • If compaction fires too early, decrease the buffer.
  • If it fires too late, increase the buffer.
  • Keep the buffer smaller than the context window. A buffer as large as the window leaves a threshold of one token and no useful request budget.

For example, with a 128000-token window, increasing the buffer from 13000 to 16000 moves the trigger from 115000 to 112000 estimated tokens: compaction happens earlier.

COMPACT_KEEP_RECENT

The default is 12 non-system messages. The agent moves the split back to a user message so it keeps complete recent turns, including their tool calls and results. It can therefore keep more than 12 messages.

Compaction requires at least COMPACT_KEEP_RECENT + 4 non-system messages and an older complete turn to summarize. A large latest turn cannot be shortened by this mechanism.

The active history becomes:

system message
summary of older turns
assistant acknowledgement
recent complete turns

Increasing COMPACT_KEEP_RECENT retains more original detail but frees less space. Decreasing it gives the summary more responsibility for earlier context. The separate PROMPT_MAX_HISTORY_MESSAGES and PROMPT_MAX_HISTORY_CHARS settings also bound history loaded from the database.

Example configurations

These examples assume the model has no catalog context_window override. Restart the server after changing environment settings.

Small local model (4 K context)

COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_LOCAL=4096
COMPACT_BUFFER_TOKENS=1024
COMPACT_KEEP_RECENT=6

Threshold: 3072 estimated tokens. Large tool schemas or a single large attachment can still exceed this budget; reducing the buffer is not a substitute for a model with enough context.

OpenRouter model (128 K context)

COMPACT_ENABLED=true
COMPACT_CONTEXT_WINDOW_OPENROUTER=128000
COMPACT_BUFFER_TOKENS=16000
COMPACT_KEEP_RECENT=12

Threshold: 112000 estimated tokens.

Circuit breaker

A failed summarization request or empty summary increments the failure count. After COMPACT_MAX_FAILURES failures (default 3), the agent stops attempting compaction for that run. A successful summary resets the count.

The failure count and a five-minute retry cooldown are stored with the chat's compaction checkpoint. Later turns reload that state. Restarting the server does not bypass a live cooldown; a new turn after it expires can retry.

A summary and its history boundary are also persisted, so subsequent turns can reuse the compacted context. Compaction does not delete the original messages.

What the compaction summary contains

The agent asks the current chat model to summarize the request, technical context, files, errors, completed work, and pending work. That call has no tools. If a <summary> block is returned, its content is used; otherwise a nonempty response is accepted as the summary.

The summarizer receives a shortened text view of older turns: up to 3000 characters per message, 300 characters per tool-call argument string, and 600 characters per tool result. Reasoning fields and image contents are not included. Preserve critical requirements in recent messages when exact details matter.

Disabling compaction

COMPACT_ENABLED=false

Restart the server to stop new automatic summaries. Existing checkpoints still load, and request/history budgets still apply.

The implementation is in src/numi_chat/agent/runtime.py (_should_compact, _compact_conversation, _build_llm_request) and src/numi_chat/chat/context.py (load_history, save_compaction). See Message Flow and Persistence for the relationship between stored messages and model context.