Skip to content

Architecture Overview

Numi Chat runs a FastAPI server that serves the web UI, persists conversations, and coordinates model requests and tool execution. The browser uses HTTP for account and chat management, and WebSocket JSON events for live streaming turns.

System at a Glance

flowchart TD
    Browser[Browser] -->|HTTP and WebSocket| Web[FastAPI routes]
    Web --> Services[Chat and account services]
    Services --> Agent[Agent runtime]
    Agent --> Adapters[LLM protocol adapters]
    Adapters --> Providers[Configured model endpoints]
    Agent --> Tools[Tool registry and executor]
    Tools --> External[Tool APIs and sandboxed code]
    Services --> Data[Database and chat files]
    Agent --> Data

The application process owns active turns, worker pools, and stream buffers. Chat messages and metadata live in SQLite (chats.db by default), while uploads and generated files live under state/. Because the producer thread runs on the server, an active turn continues even if the browser tab disconnects or reloads, and buffered events are replayed when the tab reconnects.

The Agent Loop: Heart of the System

Agent.run_stream() in src/numi_chat/agent/runtime.py drives a user turn. It persists the user message, then delegates the turn loop to run_turn() in src/numi_chat/agent_core/loop.py (see Agent Core):

  1. Refresh chat settings, validate attachment references, and save the user message (recording its <current_time> send-time note).
  2. Assemble the request from the system prompt, opening user context (<user_context>), bounded chat history, attachments, and active tool schemas.
  3. When auto-compaction is enabled and the estimated token count exceeds the model's threshold, summarize older turns. Apply the request context budget before calling the provider.
  4. Stream the model response through the wire adapter, accumulating answer text, reasoning traces, and tool calls.
  5. Save the assistant message. If no tool calls were emitted, emit done.
  6. Otherwise run the requested tools through ToolExecutorService, save their results in call order, and loop back to step 2.

Every turn is bounded by iteration, wall-clock time, output character, and tool-call limits (AGENT_* or DEEP_RESEARCH_*). See Message Flow and Persistence for event and persistence boundaries.

Agent Core

src/numi_chat/agent_core/ implements the core turn loop independently of the web application. It owns the state machine for model steps, partial output recovery, pre-stream retries, cancellation (TurnControl), deadlines, and step/tool/output budgets. It imports only numi_chat.llm and numi_chat.core.config, never database, service, tool, or web modules (enforced in CI by tests/agent_core/test_boundary.py).

run_turn() interacts with a TurnHost protocol for application-specific behavior: building the LLMRequest, recording a completed step, running tools, and running hooks before a step or after tool execution:

  • Chat Agent (agent/runtime.py): Persists messages in SQLite, compacts history, tracks citations and research spend, and picks up queued user messages between tool batches.
  • Deep research workers (agent/research_worker.py): Reuse the core streaming and adapter layer with a lean search-and-read loop.
  • SimpleAgent (agent_core/ & sdk/): An in-memory TurnHost for standalone Python scripts and tests, replaying reasoning under the same provider rules as chat turns.

numi_chat.sdk exposes SimpleAgent, Client, @tool, and the app's standalone read-only tools (default_tools(), app_tools()):

from typing import Annotated

from pydantic import Field

from numi_chat.sdk import Client, SimpleAgent, default_tools, tool


@tool
def greet(
    name: Annotated[str, Field(description="Who to greet")],
    formal: bool = False,
) -> str:
    """Greet someone by name."""
    return f"Good day, {name}." if formal else f"Hey {name}!"


client = Client.from_catalog("z-ai/glm-5.3")
agent = SimpleAgent(client, tools=[*default_tools(), greet], max_turns=15)
print(agent.ask("What's the weather in Berlin? Then greet Erik."))
  • Client: Client.from_catalog(model_id) uses a catalog model with the app's configured provider credentials (admin-only ChatGPT subscription models are excluded). Client.openai_compatible(model, base_url=..., api_key=...) connects to any Chat Completions endpoint directly.
  • Tools: @tool builds a JSON schema from type annotations and docstrings and validates arguments at call time. app_tools([...]) selects built-in read-only tools by name (excluding chat/account-bound tools such as memory, create_file, or code_interpreter), and default_tools() returns all such tools enabled on the current installation.
  • Execution: agent.run(text) yields streaming event dictionaries (status, reasoning, content, tool_start, tool_end, done, error), while agent.ask(text) returns the final text reply. agent.abort() cancels a running turn from another thread.

Separation of Concerns

Web Layer (web/)

Routes serve HTML/static assets and expose /api/v1 and /ws/{chat_id}. Route handlers authenticate the request and verify resource ownership, then delegate all database reads, writes, and business logic to services/. tests/web/test_route_boundaries.py enforces that route files never query or mutate the database directly.

When a turn starts, services/chats/stream.py spawns a producer task and publishes events through web/connection_manager.py, whose bounded replay buffer allows browsers to disconnect and reconnect mid-turn without interrupting generation.

Source Responsibility
web/app.py FastAPI app factory, middleware, route registration, and startup migrations
web/routes/ws.py WebSocket authentication, subscription, and client command dispatch
web/deps.py Authentication and chat ownership FastAPI dependencies
services/chats/stream.py Turn producer lifecycle, optional tool gate, event publishing, title and memory triggers
web/connection_manager.py Active turn tracking, subscriber queues, reconnect replay buffers, and cancellation

(Paths in tables are relative to src/numi_chat/.)

Agent and LLM Layers

The agent uses a unified internal message format, which wire adapters in llm/adapters/ translate into OpenAI Chat Completions (openai_chat), OpenAI Responses (openai_responses), or Anthropic Messages (anthropic_messages).

Source Responsibility
agent_core/ Turn loop (run_turn), streaming/retries, Client, @tool, SimpleAgent
sdk/ Library facade combining agent_core with standalone read-only tools
agent/runtime.py Chat TurnHost (Agent): history loading, compaction, prompt assembly, tool orchestration
agent/prompt_builder.py System prompt construction and file/PDF context formatting
agent/cache_warmer.py Keeps 5-minute provider prompt caches warm while deep research waits on sub-agents
agent/context_layers.py Stored <user_context> wrappers and legacy context loading; services/memory/context.py appends immutable memory updates per user turn
agent/messages.py Strips internal application keys before LLM submission
agent/models.py Model availability, fallback resolution, and owner-scoped client construction
agent/research_worker.py Background deep research sub-agent loop and report writer
services/deep_research.py Coordinates sub-agent waves, progress events, citation rewriting, and usage rollup
services/skills.py Loads built-in (skills/) and private (state/skills/) skills, admin toggles, and sandbox mounts
llm/catalog.py Runtime model catalog validation, hot-reload snapshots, and capability metadata
llm/providers.py Provider configuration, credential checks, and cached SDK clients
llm/transport.py Transport-error and retry classification
llm/messages.py Provider-specific rules for replaying reasoning traces in history
llm/adapters/ Wire protocol payload builders and stream parsers

To preserve provider prompt caching, the opening <user_context> block is attached to the first user message without any live clock, and each user message records its own send timestamp (time_note). Stored reasoning is filtered at request time according to the target model's preserve_reasoning and provider rules (see Preserved Reasoning).

Services Layer (services/)

Services implement domain logic shared across HTTP routes, WebSocket streams, and background workers:

  • auth/: Password hashing, JWT/refresh session rotation, encrypted personal API keys, and login throttling.
  • account/: User preferences, access tiers (tiers.py), and low-level per-user records (records.py).
  • chats/: Chat CRUD, turn streaming (stream.py), titles, public share snapshots (shares.py), past_chats search, and citation indexing.
  • files/: Upload validation, PDF extraction workers, library attachment, and generated artifact storage.
  • memory/: Idle chat extraction (extractor.py), lock-free consolidation (consolidator.py), cross-chat profile aggregation (aggregator.py), prompt rendering (entries.py), and memory import.
  • models/: Runtime catalog management and background role model overrides.
  • tools/: Parallel/serial tool execution (executor.py), the pre-turn activation gate (gate.py), and sandbox helpers.
  • schedules.py: The scheduler loop started in the app lifespan. Every 30 seconds it moves due tasks to their next run, then starts each run as an ordinary turn through the same stream and producer as a queued message, with the tool activation gate skipped. One process owns the loop.
  • notifications.py: Web Push (RFC 8291 encryption, VAPID signing), the visit record, and the idle pause.
  • web/: Multi-provider web search and page extraction (providers/).

Tool System

At startup, ToolRegistry discovers BaseTool subclasses in src/numi_chat/tools/. TOOLS_ALLOWLIST, TOOLS_DENYLIST, and is_available() govern which tools register; per-chat settings and user access tiers determine which tools are active on a turn.

Within a single model step, ToolExecutorService runs concurrency-safe calls in parallel on a bounded thread pool and runs non-concurrent calls serially afterward. See Tool System Design.

Data Layer (data/)

SQLModel tables live in data/models.py, while data/db.py configures the SQLite engine and runs pending Alembic migrations on startup. chat/context.py (ChatContextService) loads bounded conversation history and compaction checkpoints and persists new messages and response versions (ChatVersion).

Browser Frontend (web/static/)

  • API & auth refresh: js/api.js wraps authenticated fetch calls and handles transparent token refresh.
  • Incremental Markdown streaming: While an answer streams, js/chat/stream.js re-renders only the active tail block so completed paragraphs keep their DOM nodes and remain selectable.
  • Scroll anchoring: js/chat/scroll.js anchors a newly sent message near the top of the viewport and lets the answer grow smoothly underneath, yielding control immediately if the user scrolls manually.
  • Offline-capable PWA: sw.js (registered by js/core/pwa.js at /sw.js) implements a network-first service worker that caches same-origin GET responses for offline reading and purges the cache on sign-out. See Install Numi as an App.

Documents Layer

documents/ handles PDF page extraction, optional local OCR (pdftoppm + tesseract), chunking, and retrieval. Uploaded files and generated artifacts are isolated per chat UUID. See Chat with PDF Documents.

Deployment and Web Provider Accounting

Numi is designed to run as a single application process backed by SQLite (either directly on the host or in one Docker container), since active turn buffers, WebSocket subscriber queues, and single-use ChatGPT refresh token locks live in process memory.

For web search and page extraction (services/web/providers/), each provider adapter reports its request cost at the HTTP boundary (services/tools/usage.py). Tool charges are stored on the tool result message's usage_json and summed with model inference usage in llm/usage.py so chat overviews display separate inference and tools cost breakdowns.