Architecture Overview¶
Numi Chat runs a FastAPI server that serves the web UI, persists conversations, and coordinates model requests and tool execution. The browser uses HTTP for account and chat management, and WebSocket JSON events for live streaming turns.
System at a Glance¶
flowchart TD
Browser[Browser] -->|HTTP and WebSocket| Web[FastAPI routes]
Web --> Services[Chat and account services]
Services --> Agent[Agent runtime]
Agent --> Adapters[LLM protocol adapters]
Adapters --> Providers[Configured model endpoints]
Agent --> Tools[Tool registry and executor]
Tools --> External[Tool APIs and sandboxed code]
Services --> Data[Database and chat files]
Agent --> Data
The application process owns active turns, worker pools, and stream buffers.
Chat messages and metadata live in SQLite (chats.db by default), while uploads
and generated files live under state/. Because the producer thread runs on the
server, an active turn continues even if the browser tab disconnects or reloads,
and buffered events are replayed when the tab reconnects.
The Agent Loop: Heart of the System¶
Agent.run_stream() in src/numi_chat/agent/runtime.py drives a user turn. It persists the user message, then delegates the turn loop to run_turn() in src/numi_chat/agent_core/loop.py (see Agent Core):
- Refresh chat settings, validate attachment references, and save the user
message (recording its
<current_time>send-time note). - Assemble the request from the system prompt, opening user context
(
<user_context>), bounded chat history, attachments, and active tool schemas. - When auto-compaction is enabled and the estimated token count exceeds the model's threshold, summarize older turns. Apply the request context budget before calling the provider.
- Stream the model response through the wire adapter, accumulating answer text, reasoning traces, and tool calls.
- Save the assistant message. If no tool calls were emitted, emit
done. - Otherwise run the requested tools through
ToolExecutorService, save their results in call order, and loop back to step 2.
Every turn is bounded by iteration, wall-clock time, output character, and tool-call limits (AGENT_* or DEEP_RESEARCH_*). See Message Flow and Persistence for event and persistence boundaries.
Agent Core¶
src/numi_chat/agent_core/ implements the core turn loop independently of the
web application. It owns the state machine for model steps, partial output
recovery, pre-stream retries, cancellation (TurnControl), deadlines, and
step/tool/output budgets. It imports only numi_chat.llm and
numi_chat.core.config, never database, service, tool, or web modules (enforced
in CI by tests/agent_core/test_boundary.py).
run_turn() interacts with a TurnHost protocol for application-specific
behavior: building the LLMRequest, recording a completed step, running tools,
and running hooks before a step or after tool execution:
- Chat
Agent(agent/runtime.py): Persists messages in SQLite, compacts history, tracks citations and research spend, and picks up queued user messages between tool batches. - Deep research workers (
agent/research_worker.py): Reuse the core streaming and adapter layer with a lean search-and-read loop. SimpleAgent(agent_core/&sdk/): An in-memoryTurnHostfor standalone Python scripts and tests, replaying reasoning under the same provider rules as chat turns.
numi_chat.sdk exposes SimpleAgent, Client, @tool, and the app's
standalone read-only tools (default_tools(), app_tools()):
from typing import Annotated
from pydantic import Field
from numi_chat.sdk import Client, SimpleAgent, default_tools, tool
@tool
def greet(
name: Annotated[str, Field(description="Who to greet")],
formal: bool = False,
) -> str:
"""Greet someone by name."""
return f"Good day, {name}." if formal else f"Hey {name}!"
client = Client.from_catalog("z-ai/glm-5.3")
agent = SimpleAgent(client, tools=[*default_tools(), greet], max_turns=15)
print(agent.ask("What's the weather in Berlin? Then greet Erik."))
Client:Client.from_catalog(model_id)uses a catalog model with the app's configured provider credentials (admin-only ChatGPT subscription models are excluded).Client.openai_compatible(model, base_url=..., api_key=...)connects to any Chat Completions endpoint directly.- Tools:
@toolbuilds a JSON schema from type annotations and docstrings and validates arguments at call time.app_tools([...])selects built-in read-only tools by name (excluding chat/account-bound tools such as memory,create_file, orcode_interpreter), anddefault_tools()returns all such tools enabled on the current installation. - Execution:
agent.run(text)yields streaming event dictionaries (status,reasoning,content,tool_start,tool_end,done,error), whileagent.ask(text)returns the final text reply.agent.abort()cancels a running turn from another thread.
Separation of Concerns¶
Web Layer (web/)¶
Routes serve HTML/static assets and expose /api/v1 and /ws/{chat_id}. Route
handlers authenticate the request and verify resource ownership, then delegate
all database reads, writes, and business logic to services/.
tests/web/test_route_boundaries.py enforces that route files never query or
mutate the database directly.
When a turn starts, services/chats/stream.py spawns a producer task and
publishes events through web/connection_manager.py, whose bounded replay
buffer allows browsers to disconnect and reconnect mid-turn without interrupting
generation.
| Source | Responsibility |
|---|---|
web/app.py |
FastAPI app factory, middleware, route registration, and startup migrations |
web/routes/ws.py |
WebSocket authentication, subscription, and client command dispatch |
web/deps.py |
Authentication and chat ownership FastAPI dependencies |
services/chats/stream.py |
Turn producer lifecycle, optional tool gate, event publishing, title and memory triggers |
web/connection_manager.py |
Active turn tracking, subscriber queues, reconnect replay buffers, and cancellation |
(Paths in tables are relative to src/numi_chat/.)
Agent and LLM Layers¶
The agent uses a unified internal message format, which wire adapters in
llm/adapters/ translate into OpenAI Chat Completions (openai_chat), OpenAI
Responses (openai_responses), or Anthropic Messages (anthropic_messages).
| Source | Responsibility |
|---|---|
agent_core/ |
Turn loop (run_turn), streaming/retries, Client, @tool, SimpleAgent |
sdk/ |
Library facade combining agent_core with standalone read-only tools |
agent/runtime.py |
Chat TurnHost (Agent): history loading, compaction, prompt assembly, tool orchestration |
agent/prompt_builder.py |
System prompt construction and file/PDF context formatting |
agent/cache_warmer.py |
Keeps 5-minute provider prompt caches warm while deep research waits on sub-agents |
agent/context_layers.py |
Stored <user_context> wrappers and legacy context loading; services/memory/context.py appends immutable memory updates per user turn |
agent/messages.py |
Strips internal application keys before LLM submission |
agent/models.py |
Model availability, fallback resolution, and owner-scoped client construction |
agent/research_worker.py |
Background deep research sub-agent loop and report writer |
services/deep_research.py |
Coordinates sub-agent waves, progress events, citation rewriting, and usage rollup |
services/skills.py |
Loads built-in (skills/) and private (state/skills/) skills, admin toggles, and sandbox mounts |
llm/catalog.py |
Runtime model catalog validation, hot-reload snapshots, and capability metadata |
llm/providers.py |
Provider configuration, credential checks, and cached SDK clients |
llm/transport.py |
Transport-error and retry classification |
llm/messages.py |
Provider-specific rules for replaying reasoning traces in history |
llm/adapters/ |
Wire protocol payload builders and stream parsers |
To preserve provider prompt caching, the opening <user_context> block is attached to the first user message without any live clock, and each user message records its own send timestamp (time_note). Stored reasoning is filtered at request time according to the target model's preserve_reasoning and provider rules (see Preserved Reasoning).
Services Layer (services/)¶
Services implement domain logic shared across HTTP routes, WebSocket streams, and background workers:
auth/: Password hashing, JWT/refresh session rotation, encrypted personal API keys, and login throttling.account/: User preferences, access tiers (tiers.py), and low-level per-user records (records.py).chats/: Chat CRUD, turn streaming (stream.py), titles, public share snapshots (shares.py),past_chatssearch, and citation indexing.files/: Upload validation, PDF extraction workers, library attachment, and generated artifact storage.memory/: Idle chat extraction (extractor.py), lock-free consolidation (consolidator.py), cross-chat profile aggregation (aggregator.py), prompt rendering (entries.py), and memory import.models/: Runtime catalog management and background role model overrides.tools/: Parallel/serial tool execution (executor.py), the pre-turn activation gate (gate.py), and sandbox helpers.schedules.py: The scheduler loop started in the app lifespan. Every 30 seconds it moves due tasks to their next run, then starts each run as an ordinary turn through the same stream and producer as a queued message, with the tool activation gate skipped. One process owns the loop.notifications.py: Web Push (RFC 8291 encryption, VAPID signing), the visit record, and the idle pause.web/: Multi-provider web search and page extraction (providers/).
Tool System¶
At startup, ToolRegistry discovers BaseTool subclasses in
src/numi_chat/tools/. TOOLS_ALLOWLIST, TOOLS_DENYLIST, and
is_available() govern which tools register; per-chat settings and user access
tiers determine which tools are active on a turn.
Within a single model step, ToolExecutorService runs concurrency-safe calls in parallel on a bounded thread pool and runs non-concurrent calls serially afterward. See Tool System Design.
Data Layer (data/)¶
SQLModel tables live in data/models.py, while data/db.py configures the
SQLite engine and runs pending Alembic migrations on startup. chat/context.py
(ChatContextService) loads bounded conversation history and compaction
checkpoints and persists new messages and response versions (ChatVersion).
Browser Frontend (web/static/)¶
- API & auth refresh:
js/api.jswraps authenticatedfetchcalls and handles transparent token refresh. - Incremental Markdown streaming: While an answer streams,
js/chat/stream.jsre-renders only the active tail block so completed paragraphs keep their DOM nodes and remain selectable. - Scroll anchoring:
js/chat/scroll.jsanchors a newly sent message near the top of the viewport and lets the answer grow smoothly underneath, yielding control immediately if the user scrolls manually. - Offline-capable PWA:
sw.js(registered byjs/core/pwa.jsat/sw.js) implements a network-first service worker that caches same-origin GET responses for offline reading and purges the cache on sign-out. See Install Numi as an App.
Documents Layer¶
documents/ handles PDF page extraction, optional local OCR (pdftoppm + tesseract), chunking, and retrieval. Uploaded files and generated artifacts are isolated per chat UUID. See Chat with PDF Documents.
Deployment and Web Provider Accounting¶
Numi is designed to run as a single application process backed by SQLite (either directly on the host or in one Docker container), since active turn buffers, WebSocket subscriber queues, and single-use ChatGPT refresh token locks live in process memory.
For web search and page extraction (services/web/providers/), each provider
adapter reports its request cost at the HTTP boundary
(services/tools/usage.py). Tool charges are stored on the tool result
message's usage_json and summed with model inference usage in llm/usage.py
so chat overviews display separate inference and tools cost breakdowns.