Skip to content

Running Tests

Install the development dependencies with make install-dev first; see Development Setup if you are working in a fresh checkout.

Quick Start

Run the same checks that execute in the main CI workflow:

make check
npm test
uv run pytest -m "not network"
  • make check runs formatting, lint, dead-code, and core-path complexity checks without modifying files.
  • npm test runs the browser behavior tests in tests/js/*.cjs using Node's built-in test runner.
  • uv run pytest -m "not network" runs the Python test suite while skipping tests that make live external network calls.

make test runs pytest without -m "not network" and does not run the JavaScript suite.

Running Focused Tests

Python tests mirror the package layout under src/numi_chat/ (tests/agent/, tests/services/, tests/tools/, tests/web/, and so on), while JavaScript tests live in tests/js/. Start with the test file closest to your change:

uv run pytest tests/agent/test_agent_runtime.py
uv run pytest tests/tools/test_tool_registry.py
uv run pytest tests/services/test_skills.py tests/skills
uv run pytest tests/web/test_security.py::TestPasswordSecurity::test_password_hash_uniqueness
node --test tests/js/test_chat_menu.cjs

Add -x -v to stop on the first failure with verbose test names, or --pdb to drop into the debugger on failure.

The project defines one custom pytest marker: network. Marker names are checked strictly (--strict-markers), so select tests by file path or with -k keyword expressions.

Network and sandbox checks

To run tests that contact external services:

uv run pytest -m network

To run the full Deno/Pyodide package and office-file round-trip test:

NUMI_TEST_SANDBOX=1 uv run pytest tests/tools/test_code_interpreter.py -k office_files_roundtrip

This downloads Pyodide packages into the sandbox cache, verifies office file generation and re-reading, and checks that outbound networking is blocked.

Test Structure

Area Key test files
Agent and provider behavior test_agent_runtime.py, test_turn_golden.py, test_agent_messages.py, test_llm_adapters.py
Agent core and SDK tests/agent_core/ (streaming, Client, @tool, SimpleAgent, import boundary), tests/sdk/
Chat API and authentication test_api.py, test_auth.py, test_auth_sessions.py
Database and migrations test_migrations.py, test_repositories.py, test_deletion_integrity.py
Tools test_tool_registry.py, test_tool_executor.py, test_code_interpreter.py
Browser behavior test_chat_menu.cjs, test_chat_recovery.cjs, test_memory_profile.cjs, test_frontpage.cjs, test_i18n.cjs

JavaScript tests run under Node against lightweight DOM doubles. When changing UI layout or styling, also verify the page in a real browser across desktop and mobile widths in both light and dark themes.

Writing Tests

Reuse the fixtures in tests/conftest.py:

Fixture Provides
in_memory_db An isolated SQLite engine with SQLModel tables and foreign keys enabled
db_session A database session bound to in_memory_db
test_client A FastAPI TestClient wired to the test database session
authenticated_client A TestClient with a registered user and Bearer token
test_user_data, test_user_data2 Registration payloads for two test accounts

tests/factories.py provides helper functions (create_test_user, create_test_chat, create_test_message, create_test_memory) for seeding state directly. Mock external HTTP or model calls with monkeypatch or unittest.mock so standard tests stay fast and offline.

tests/conftest.py sets a temporary default DB_URL via os.environ.setdefault. Do not export a production DB_URL in your shell when running pytest.

For database schema changes, test migrating a populated legacy database as well as a fresh one; see Database Migrations.

For streaming refactors, keep both retry boundaries covered: a transport failure before the first event may retry, but a failure after output must not replay it. tests/agent_core/test_streaming.py exercises both paths, including cancellation and exhausted time budgets.

Coverage

make test-cov prints missing-line coverage in the terminal. To generate a browsable HTML report:

uv run pytest -m "not network" --cov=src/numi_chat --cov-report=html

Then open htmlcov/index.html.

CI/CD Integration

.github/workflows/tests.yml restores pinned browser assets, installs Node and Python dependencies, and runs make check, npm test, and uv run pytest -m "not network" with coverage.