Running Tests¶
Install the development dependencies with make install-dev first; see
Development Setup if you are working in a fresh checkout.
Quick Start¶
Run the same checks that execute in the main CI workflow:
make checkruns formatting, lint, dead-code, and core-path complexity checks without modifying files.npm testruns the browser behavior tests intests/js/*.cjsusing Node's built-in test runner.uv run pytest -m "not network"runs the Python test suite while skipping tests that make live external network calls.
make test runs pytest without -m "not network" and does not run the
JavaScript suite.
Running Focused Tests¶
Python tests mirror the package layout under src/numi_chat/ (tests/agent/,
tests/services/, tests/tools/, tests/web/, and so on), while JavaScript
tests live in tests/js/. Start with the test file closest to your change:
uv run pytest tests/agent/test_agent_runtime.py
uv run pytest tests/tools/test_tool_registry.py
uv run pytest tests/services/test_skills.py tests/skills
uv run pytest tests/web/test_security.py::TestPasswordSecurity::test_password_hash_uniqueness
node --test tests/js/test_chat_menu.cjs
Add -x -v to stop on the first failure with verbose test names, or --pdb to
drop into the debugger on failure.
The project defines one custom pytest marker: network. Marker names are
checked strictly (--strict-markers), so select tests by file path or with -k
keyword expressions.
Network and sandbox checks¶
To run tests that contact external services:
To run the full Deno/Pyodide package and office-file round-trip test:
This downloads Pyodide packages into the sandbox cache, verifies office file generation and re-reading, and checks that outbound networking is blocked.
Test Structure¶
| Area | Key test files |
|---|---|
| Agent and provider behavior | test_agent_runtime.py, test_turn_golden.py, test_agent_messages.py, test_llm_adapters.py |
| Agent core and SDK | tests/agent_core/ (streaming, Client, @tool, SimpleAgent, import boundary), tests/sdk/ |
| Chat API and authentication | test_api.py, test_auth.py, test_auth_sessions.py |
| Database and migrations | test_migrations.py, test_repositories.py, test_deletion_integrity.py |
| Tools | test_tool_registry.py, test_tool_executor.py, test_code_interpreter.py |
| Browser behavior | test_chat_menu.cjs, test_chat_recovery.cjs, test_memory_profile.cjs, test_frontpage.cjs, test_i18n.cjs |
JavaScript tests run under Node against lightweight DOM doubles. When changing UI layout or styling, also verify the page in a real browser across desktop and mobile widths in both light and dark themes.
Writing Tests¶
Reuse the fixtures in tests/conftest.py:
| Fixture | Provides |
|---|---|
in_memory_db |
An isolated SQLite engine with SQLModel tables and foreign keys enabled |
db_session |
A database session bound to in_memory_db |
test_client |
A FastAPI TestClient wired to the test database session |
authenticated_client |
A TestClient with a registered user and Bearer token |
test_user_data, test_user_data2 |
Registration payloads for two test accounts |
tests/factories.py provides helper functions (create_test_user,
create_test_chat, create_test_message, create_test_memory) for seeding
state directly. Mock external HTTP or model calls with monkeypatch or
unittest.mock so standard tests stay fast and offline.
tests/conftest.py sets a temporary default DB_URL via
os.environ.setdefault. Do not export a production DB_URL in your shell when
running pytest.
For database schema changes, test migrating a populated legacy database as well as a fresh one; see Database Migrations.
For streaming refactors, keep both retry boundaries covered: a transport
failure before the first event may retry, but a failure after output must
not replay it. tests/agent_core/test_streaming.py exercises both paths,
including cancellation and exhausted time budgets.
Coverage¶
make test-cov prints missing-line coverage in the terminal. To generate a
browsable HTML report:
Then open htmlcov/index.html.
CI/CD Integration¶
.github/workflows/tests.yml restores pinned browser assets, installs Node and
Python dependencies, and runs make check, npm test, and
uv run pytest -m "not network" with coverage.