Add comprehensive test suites for answer key validation, API functionality, and code linting
- Introduced `test_answer_key.py` to validate the correctness of answer keys against expected responses, ensuring both false and true answers are graded appropriately. - Created `test_answer_values.py` to verify the accuracy of answer keys by recomputing values and comparing them against the provided `answers.json`. - Implemented `test_api.py` to test the HTTP API for suite management, ensuring read-only enforcement and proper handling of custom suites. - Added `test_code_lint.py` to perform static code analysis, ensuring that all linting rules are enforced without false positives on correct answers. - Developed `test_collision.py` to guard against prompt mismatches due to filename collisions across suites. - Created `test_gguf.py` to validate GGUF header parsing and model classification, including handling of malformed input. - Introduced `test_linter.py` to ensure report rendering preserves content integrity and correctly fences code. - Added `test_reporting.py` to verify report naming conventions and the accuracy of per-suite score breakdowns.
This commit is contained in:
@@ -300,6 +300,18 @@ Reports are written to `results/automated_report_<model>.md`.
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
python tests_py/run_all.py # 195 assertions, no dependencies
|
||||
cd web && npm test # 29 UI assertions
|
||||
```
|
||||
|
||||
`tests_py/test_api.py` needs a running server and skips cleanly without one.
|
||||
Details in [`tests_py/README.md`](tests_py/README.md).
|
||||
|
||||
---
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
@@ -320,6 +332,7 @@ server/ FastAPI backend wrapping the engine
|
||||
plugins/ Drop-in plugins — _skeleton.py is the starter template
|
||||
plugin_api.py Stable types plugins import
|
||||
plugin_system.py Plugin discovery and hook dispatch
|
||||
tests_py/ Python tests — python tests_py/run_all.py
|
||||
web/ React + TypeScript interface (Vite)
|
||||
models/ Drop .gguf or MLX weights here (auto-discovered)
|
||||
.core/suites/<slug>/ Built-in suites, read-only
|
||||
|
||||
Reference in New Issue
Block a user