Python tests
python tests_py/run_all.py # everything
python tests_py/run_all.py gguf key # only files matching those words
python tests_py/test_linter.py # one file directly
Exits non-zero if anything fails, so it works as a CI entry point.
No third-party dependencies. test_api.py needs a running server
(python app.py) and the requests package the LM Studio provider already
uses; it prints SKIP and passes when the server is down, so the rest stay
useful offline.
| File | Covers |
|---|---|
test_answer_values.py |
The answer keys themselves, recomputed from scratch |
test_answer_key.py |
The correctness grader, both directions |
test_api.py |
Suites API, read-only guards, run selection |
test_code_lint.py |
Static code rules, and the false positives they must not repeat |
test_collision.py |
Qualified question IDs |
test_gguf.py |
GGUF header parsing and model classification |
test_linter.py |
Report rendering |
test_lmstudio.py |
LM Studio model listing, against stubbed HTTP |
test_reporting.py |
Report naming and the per-suite breakdown |
The UI tests live separately, under web/src/**/*.test.tsx — run them with
cd web && npm test.
Why these are scripts rather than unittest cases
They exercise a diagnostic tool, so the output is read as much as the exit
code. test_code_lint.py is a table of which rules fire on which input, and
that is worth more when debugging a false positive than a row of dots.
run_all.py aggregates exit codes for the times you only want a verdict.
Each file runs in its own process. Several load plugins directly, which would
otherwise collide in sys.modules.
test_answer_values.py is the one to keep honest
The other files test code. This one tests data — it recomputes every
numeric answer key with exact arithmetic and brute-force search, then asserts
the shipped answers.json agrees.
That matters because a wrong key fails correct work with total confidence, which is the exact failure the answer-key grader exists to remove. It earned its place immediately: the Winograd referent had been recorded backwards, and shipping it would have marked every correct answer wrong.
If you add or edit a key, add the computation here too. A key nobody can re-derive is a liability.