Running Tests¶

Install development dependencies with make install-dev; see Development Setup if this is a new checkout.

Quick Start¶

Run the checks used by the main CI test job:

make check
npm test
uv run pytest -m "not network"

make check checks code without editing it. npm test runs the tests/*.cjs files with Node's built-in test runner. The pytest command excludes tests that call external services.

make test runs pytest without that exclusion. It does not run the JavaScript tests.

Running Tests¶

Start with the file closest to your change:

uv run pytest tests/test_agent_runtime.py
uv run pytest tests/test_tool_registry.py
uv run pytest tests/test_security.py::TestPasswordSecurity::test_password_hash_uniqueness
node --test tests/test_chat_menu.cjs

To investigate a failure, add -x -v to stop at the first failing test and show its name, or --pdb to enter the Python debugger.

The project registers one custom pytest marker: network. Marker names are checked strictly; unit, integration, security, and slow are not defined. Select by file or use -k for a test-name filter.

Network and sandbox checks¶

Run external-service tests explicitly:

uv run pytest -m network

The real code-interpreter package test also requires an opt-in flag and Deno:

NUMI_TEST_SANDBOX=1 uv run pytest tests/test_code_interpreter.py -k office_files_roundtrip

This downloads sandbox packages, then checks office-file round trips and blocked networking during execution. It is skipped without the flag.

Test Structure¶

Area	Existing tests
Agent and provider behavior	`test_agent_runtime.py`, `test_agent_messages.py`, `test_llm_adapters.py`
Chat API and authentication	`test_api.py`, `test_auth.py`, `test_auth_sessions.py`
Database changes	`test_migrations.py`, `test_repositories.py`, `test_deletion_integrity.py`
Tools	`test_tool_registry.py`, `test_tool_executor.py`, `test_code_interpreter.py`
Browser behavior	`test_chat_menu.cjs`, `test_chat_recovery.cjs`, `test_memory_profile.cjs`, `test_frontpage.cjs`

The JavaScript tests exercise behavior with Node and test doubles. For a UI change, also check the rendered app in a browser.

Writing Tests¶

Reuse the patterns in the nearest test file. Shared fixtures in tests/conftest.py include:

Fixture	Provides
`in_memory_db`	A SQLite engine with the SQLModel tables and foreign keys enabled
`db_session`	A session backed by that in-memory database
`test_client`	A FastAPI client using the test database dependency
`authenticated_client`	The test client with a registered user and Bearer token
`test_user_data`, `test_user_data2`	Registration payloads for two users

tests/factories.py contains create_test_user, create_test_chat, create_test_message, and create_test_memory. These are helper functions, not fixtures. Use monkeypatch or unittest.mock to replace external calls in tests that should run offline.

The test configuration sets a temporary default database URL with os.environ.setdefault. An already-exported DB_URL takes precedence: do not run tests with your production database URL exported. API fixtures override the session, but not every startup or migration path uses that override.

For schema changes, test an upgrade from populated legacy tables as well as an empty database. See Database Migrations.

Coverage¶

make test-cov prints missing-line coverage in the terminal. To generate an HTML report:

uv run pytest -m "not network" --cov=src/numi_chat --cov-report=html

Open htmlcov/index.html after the command finishes.

CI/CD Integration¶

The current workflow is .github/workflows/tests.yml. Its test job restores browser assets, installs Node dependencies, then runs make check, npm test, and pytest with -m "not network" and coverage.

make ci-local runs make check and pytest -q; it does not include npm test and does not exclude network tests. Use the commands at the top of this page to match CI's test selection.