Add comprehensive test suites for answer key validation, API functionality, and code linting

- Introduced `test_answer_key.py` to validate the correctness of answer keys against expected responses, ensuring both false and true answers are graded appropriately.
- Created `test_answer_values.py` to verify the accuracy of answer keys by recomputing values and comparing them against the provided `answers.json`.
- Implemented `test_api.py` to test the HTTP API for suite management, ensuring read-only enforcement and proper handling of custom suites.
- Added `test_code_lint.py` to perform static code analysis, ensuring that all linting rules are enforced without false positives on correct answers.
- Developed `test_collision.py` to guard against prompt mismatches due to filename collisions across suites.
- Created `test_gguf.py` to validate GGUF header parsing and model classification, including handling of malformed input.
- Introduced `test_linter.py` to ensure report rendering preserves content integrity and correctly fences code.
- Added `test_reporting.py` to verify report naming conventions and the accuracy of per-suite score breakdowns.
This commit is contained in:
Netherwarlord
2026-07-28 20:26:56 -04:00
parent 7a81468925
commit f77f0af33a
12 changed files with 1122 additions and 0 deletions
+13
View File
@@ -300,6 +300,18 @@ Reports are written to `results/automated_report_<model>.md`.
---
## Tests
```bash
python tests_py/run_all.py # 195 assertions, no dependencies
cd web && npm test # 29 UI assertions
```
`tests_py/test_api.py` needs a running server and skips cleanly without one.
Details in [`tests_py/README.md`](tests_py/README.md).
---
## Project structure
```
@@ -320,6 +332,7 @@ server/ FastAPI backend wrapping the engine
plugins/ Drop-in plugins — _skeleton.py is the starter template
plugin_api.py Stable types plugins import
plugin_system.py Plugin discovery and hook dispatch
tests_py/ Python tests — python tests_py/run_all.py
web/ React + TypeScript interface (Vite)
models/ Drop .gguf or MLX weights here (auto-discovered)
.core/suites/<slug>/ Built-in suites, read-only