Architecture Overview

Numi Chat runs a FastAPI server that serves the web UI, stores chats, and coordinates model requests and tools. The browser uses HTTP for account and chat management, and WebSocket JSON events for live conversation turns.

System at a Glance

flowchart TD
    Browser[Browser] -->|HTTP and WebSocket| Web[FastAPI routes]
    Web --> Services[Chat and account services]
    Services --> Agent[Agent runtime]
    Agent --> Adapters[LLM protocol adapters]
    Adapters --> Providers[Configured model endpoints]
    Agent --> Tools[Tool registry and executor]
    Tools --> External[Tool APIs and sandboxed code]
    Services --> Data[Database and chat files]
    Agent --> Data

The application process owns active turns, worker pools, and stream buffers. Chat messages and metadata live in SQLite by default; uploads and generated files live on disk. A browser disconnect and a server restart therefore have different effects: a turn can continue after the browser disconnects, but its in-memory execution state does not survive a process restart.

The Agent Loop: Heart of the System

Agent.run_stream() in src/numi_chat/agent/runtime.py drives a user turn:

1. Refresh chat settings, validate attachment references, and save the user message.
2. Build the request from the system prompt, current user context, bounded chat history, attachments, and active tool schemas.
3. If enabled and needed, summarize older turns. Apply the request context budget before sending anything to the model.
4. Stream the model response through the adapter, collecting answer text, reasoning, and tool calls.
5. Save the assistant message. If it has no tool calls, emit done.
6. Otherwise execute the requested tools, save their results, and return to the model with those results.

The loop has iteration, time, output, and tool-call limits. Cancellation or an error ends the turn; completion is not a promise that the user's task succeeded. See Message Flow and Persistence for the event and save boundaries.

Separation of Concerns

Web Layer (web/)

Routes serve pages and expose APIs. They use authentication and ownership checks before handing chat operations to services. The WebSocket route validates incoming payloads and subscribes the browser to the chat's stream.

services/chat_stream.py starts a producer thread for a turn. It forwards agent events through web/connection_manager.py, whose bounded buffers support reconnecting to an active stream. The browser connection is a subscriber rather than the owner of the model request.

Source	Responsibility
web/app.py	Application startup, middleware, routes, database migration startup
web/routes/ws.py	WebSocket connection and message handling
web/deps.py	Authentication and chat ownership dependencies
services/chat_stream.py	Turn production, event forwarding, title and memory follow-up
web/connection_manager.py	Active turns, subscriber queues, replay buffers, cancellation

Paths in the tables are relative to src/numi_chat/.

Agent and LLM Layers

The agent uses one internal message representation. Adapters translate that representation into the selected wire protocol: OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages.

The catalog distinguishes the provider used for credentials and routing from wire_api, the protocol, and api_model, the upstream model identifier. An OpenRouter model whose name starts with openai/ still uses OpenRouter credentials. See Model Catalog.

Source	Responsibility
agent/runtime.py	Agent loop, request budgets, compaction, tool coordination
agent/prompt_builder.py	System prompt and file/document context
agent/context_layers.py	Current date, user preferences, and stored memory context
agent/messages.py	Removal of internal metadata and incompatible reasoning fields
agent/models.py	Provider availability, fallback selection, client construction
llm/catalog.py	Catalog validation, snapshots, and model metadata
llm/adapters/	Protocol payloads and stream parsing

Dynamic user context is added while preparing a request, at the first user message. It is not an extra permanent transcript message. Stored reasoning is also filtered at request time; generation effort and replay are separate controls. See Preserved Reasoning.

Services Layer (services/)

Services hold operations shared by routes and agent code, including chat updates, access checks, tool execution, artifacts, and memory processing. They are application modules, not separately deployed services. Some use FastAPI errors or SQLModel directly; this is not a strict framework-independent layer boundary.

Tool System

The registry discovers installed Python tool modules. Startup configuration controls which tools register; chat settings select active custom tools from that set. The executor checks the active set again before invocation.

Concurrency-safe calls run in a shared thread pool. Calls marked unsafe for concurrency run serially after the parallel calls in that model step. See Tool System Design for selection and execution boundaries.

Data Layer (data/)

SQLModel models are defined in data/models.py. data/db.py configures the engine and runs Alembic migrations at startup. Repositories centralize many queries, while services also perform direct database operations.

chat/context.py bridges stored messages and the agent's history. It loads recent complete turns within configured budgets and restores compaction checkpoints. The transcript stored in the database can therefore be longer than the history sent to the model.

Documents Layer

documents/ handles extraction, indexing, and retrieval for document uploads. The upload routes and artifact services manage chat-owned files; the prompt builder and message formatter select what is included with a request.

Attachments are not unrestricted filesystem access. Their inclusion depends on the chat, input modality support, processing status, and size/context limits. See Set Up Document Processing.

Deployment and limits

The normal deployment is one application process, either local or in Docker. In-process turn ownership and stream buffers are not shared between workers. Multiple processes reading the same database do not automatically share active WebSocket turns or tool execution state.

SQLite keeps deployment small but has limited write concurrency. Model and tool calls may leave the host for configured services; self-hosting the UI does not make those requests local. The catalog, database, and chat files also have different persistence paths, so a deployment needs to retain each one it uses.

Use Run with Docker for deployment steps and Privacy and Self-Hosting for data boundaries.