Skip to content

Message Flow and Persistence

Numi Chat maintains three distinct representations of a conversation: rows persisted in SQLite (Message), the filtered message list prepared for an LLM request (Agent.messages), and real-time JSON events streamed over /ws/{chat_id}.

Message Lifecycle

  1. WebSocket submission: The browser sends a user message (with a client-generated request_id and any attachment filenames) over /ws/{chat_id}. web/routes/ws.py verifies authentication, chat ownership, payload limits, and turn concurrency.
  2. Optional tool activation gate: Before persisting the message, services/chats/stream.py asks agent/tool_group_router.py whether any disabled optional tool groups strongly match the user's prompt. If so, it broadcasts a tool_gate event and waits briefly (DECISIONS_TOOL_GATE_TIMEOUT_SECONDS) for the user to accept or decline before continuing. Stopping while the card is open closes it without recording a decline and cancels the turn before the message is saved.
  3. Agent initialization & user message save: The producer creates an Agent, loads the chat's active model, settings, recent history, and latest compaction checkpoint, and saves the user message to SQLite. An accepted event returns the saved message and request_id to the browser.
  4. Streaming model step: The selected wire adapter streams reasoning, content, and tool_preparing events to the browser in real time.
  5. Assistant step persistence: When the model step finishes, the agent saves the assistant message (including any tool_calls, reasoning_content, reasoning_details_json, and usage_json) before executing tools.
  6. Tool execution: ToolExecutorService streams tool_start and tool_end events as tools finish, then saves the tool result messages in their original call order and loops back for the next model step.
  7. Turn completion: Once the model produces a final reply with no further tool calls, the agent emits done (with message_id, citation_sources, and aggregated usage), triggers background title generation or idle memory scheduling, and closes the turn with turn_end.

Message Structure (Message table)

Column Purpose
role, content Message role (system, user, assistant, tool) and primary text
content_parts_json Multimodal attachment references, the file list and PDF text first announced with the message (context_note), append-only model context (memory_note), recorded <current_time> (time_note), and request_meta (request_id, mode)
reasoning_content Plain-text reasoning trace or summary for UI display and text-based replay
reasoning_details_json Structured provider reasoning blocks (Anthropic signed thinking, OpenRouter details, OpenAI encrypted reasoning)
tool_calls_json Assistant function calls (id, name, arguments)
tool_call_id Pairs a role="tool" message with its assistant tool_call
citation_sources_json Deduplicated [web:N] source metadata for citation chips
usage_json Token counts and cost for an assistant step or metered tool call
chat_id, created_at, deleted_at Chat foreign key, timestamp, and soft-deletion marker

See Database Schema Reference for the complete table definition.

Preparing Messages for the Model

Before each LLM call, the agent transforms the loaded history into a provider-ready payload:

  1. Opening session context (_first_user_context): Injects <active_model>, <optional_tools>, and <available_skills> at the first user message without mutating stored database rows.
  2. Per-turn attachments and timestamps: Reconstructs permitted attachment text/images and replays each user message's stored context_note (new uploads and PDF text), append-only memory_note updates and time_note (<current_time>... UTC</current_time>) so the prompt prefix stays byte-identical across turns. The context and memory notes are model-only: they are not part of message payloads and never enter a share snapshot. Only the owner's chat info panel summarizes what the latest memory note holds.
  3. Reasoning & internal field filter (prepare_messages): Strips internal app keys (_db_id, additional_data) and filters reasoning_content and reasoning_details based on the target model's provider, supports_reasoning, and preserve_reasoning settings (see Preserved Reasoning).
  4. Context budget & compaction: Summarizes older turns if auto-compaction is enabled and the quad-token estimate exceeds the model's threshold, or drops older complete turns if needed to fit within the history budget.
  5. Wire adapter conversion: Converts the prepared messages and active tool schemas into Chat Completions, Responses, or Anthropic Messages format.

Chat History Loading & Compaction Checkpoints

ChatContextService.load_history() in src/numi_chat/chat/context.py loads the system message, the latest valid ChatCompaction summary (if any), and non-deleted messages from the active response branch (ChatVersion):

  • History loading is bounded by PROMPT_MAX_HISTORY_MESSAGES (default 1,000) and a character budget of max(PROMPT_MAX_HISTORY_CHARS, compact_threshold * 3) so models with large context windows can use their full capacity.
  • Complete user -> assistant -> tool groups are kept together so a request never starts with an orphaned tool result.
  • When older turns are omitted by the budget, the runtime inserts a brief notice so the model knows earlier messages were truncated.

WebSocket Streaming & Reconnect Replay

ConnectionManager (src/numi_chat/web/connection_manager.py) broadcasts JSON events with monotonic event_id and turn_id identifiers:

Event type Meaning
turn_start Turn producer started
tool_gate Pre-turn router is offering optional tool groups for activation
tool_gate_closed Pre-turn tool activation prompt was answered, timed out, or cancelled
accepted User message was saved to SQLite (request_id, message)
reasoning, content Incremental reasoning or answer text delta
tool_preparing Tool call arguments are streaming from the model
tool_start, tool_end Tool execution started / finished with output
research_agent Live status update from a Deep Research sub-agent
done Assistant step or final reply saved (message_id, citation_sources, usage)
error Turn stopped due to an error, cancellation, or budget limit
turn_end Turn producer finished
queued_messages Updated list of pending queued inputs (chatqueuedmessage)
title_update Generated or updated sidebar chat title

See API Reference for full event schemas.

If a browser tab refreshes or reconnects while a turn is running, it sends its last seen turn_id and event_id, and ConnectionManager replays the buffered events (coalescing text deltas and keeping the latest state for snapshot_key events like research_agent and tool_gate). The snapshot limits count coalesced text rather than charging a separate event envelope for every streamed chunk, so many small chunks do not prematurely cancel an otherwise bounded reply.

Queued Input (chatqueuedmessage)

Messages submitted while a turn is already running are stored in chatqueuedmessage outside the main transcript:

  • next_step continuation: Accepted only at a safe boundary after all tool results in the current batch (and any active deep research wave) have been persisted. The agent emits an intermediate done event, persists the queued user message (accepted with continuation: true), and continues the turn loop without resetting turn budgets.
  • Post-turn drain: Any remaining queued messages are drained in order after the final answer completes.
  • Crash & error safety: Queued inputs stay in chatqueuedmessage until accepted and saved as a Message row. If a turn is cancelled or errors out, unaccepted queued messages remain above the composer so the user can click Send now or edit/delete them.

Usage Accounting

Both LLM inference calls and metered web search/fetch calls record usage on their respective Message rows (usage_json):

  • OpenRouter reports the real cost of each request. For other providers the cost is an estimate from the model's list prices (input_price, cached_input_price, output_price in the model catalog) and the token counts the provider reports; a model without prices records no cost. ChatGPT plan models are estimated the same way from public API prices. Estimates are flagged estimated and list_price and shown with ≈.
  • Tool calls store their web API cost on the role="tool" message's usage_json before any output truncation or citation deduplication.
  • In Deep Research mode, research_poll attaches the sub-agents' accumulated token and tool spend onto the poll tool message's usage_json (and any unbilled spend at turn exit is recorded on a closing assistant record).
  • The done event and saved chat overview sum assistant and tool usage across the turn, breaking out inference and tools subtotals while stripping all usage metadata from model-history replay, memory extraction, and public share snapshots.