78 lines
4.0 KiB
Markdown
78 lines
4.0 KiB
Markdown
# LLM Automated Test Suite
|
||
|
||
This project automates a battery of diagnostic prompts against local or remote language models via a modular core runner and an optional GUI.
|
||
|
||
## Prerequisites
|
||
|
||
- Python 3.10 or newer (tkinter included with the standard library on macOS/Linux; install `python3-tk` on Debian-based systems if needed).
|
||
- Install dependencies:
|
||
|
||
```bash
|
||
python -m pip install -r requirements.txt
|
||
```
|
||
|
||
## Project Layout
|
||
|
||
- `.core/` – shared logic modules, templates, engines, and transient workspace used by both CLI and GUI.
|
||
- `config.py` – path/setting helpers.
|
||
- `prompts.py` – prompt discovery from `tests/`.
|
||
- `providers/` – provider adapters (LM Studio and the native Local Engine runtime).
|
||
- `reporting.py` – markdown templating, code-fence hygiene, report summaries.
|
||
- `runner.py` – orchestrates full test runs with progress callbacks.
|
||
- `engine_loader.py` – detects the host platform/GPU and loads the appropriate runtime.
|
||
- `.engine/.<architecture>/<version>.py` – runtime implementations per hardware family (Apple Silicon ↦ MLX, CUDA ↦ llama.cpp, ROCm ↦ llama.cpp, CPU fallback).
|
||
- `.temp/` – ephemeral staging for rendered blocks.
|
||
- `templates/test-block.md` – markdown template for each test result.
|
||
- `models/` – drop-in directory for local `.gguf` (and MLX-compatible) weights discovered by the Local Engine provider.
|
||
- `tests/` – one prompt per `.txt` file (editable via the GUI or any editor).
|
||
- `results/` – generated markdown reports.
|
||
- `auto-test.py` – CLI entrypoint that runs the default provider/model.
|
||
- `index.py` – Tkinter GUI for provider/model selection, running tests, and opening prompts/results.
|
||
|
||
## Running the CLI
|
||
|
||
```bash
|
||
python auto-test.py
|
||
```
|
||
|
||
The CLI selects the first available model for the default provider (LM Studio by default) and streams progress to stdout. Reports are saved in `results/automated_report_<model>.md`.
|
||
|
||
To switch to the Local Engine runtime, set `AUTO_TEST_PROVIDER="Local Engine"` (or choose it in the GUI) and ensure your models are available in the `models/` directory or any path listed in `LOCAL_LLM_PATHS`.
|
||
|
||
## Using the GUI
|
||
|
||
```bash
|
||
python index.py
|
||
```
|
||
|
||
GUI features:
|
||
|
||
- Provider dropdown with automatic model discovery.
|
||
- Model dropdown populated per provider.
|
||
- Adjustable sampling temperature.
|
||
- Buttons to run tests, refresh models, open the `tests/` folder, open the `results/` folder, and open the latest report.
|
||
- Live log of test progress and completion status.
|
||
- Settings menu → “Configure Model Paths…” to manage additional folders scanned by the Local Engine provider (persisted in `.core/user_settings.json`).
|
||
|
||
## Configuration
|
||
|
||
Environment variables:
|
||
|
||
- `LM_STUDIO_BASE_URL` – override the default `http://localhost:1234` endpoint for the LM Studio provider.
|
||
- `AUTO_TEST_TEMPERATURE` – default sampling temperature for test runs.
|
||
- `AUTO_TEST_PROVIDER` – default provider name for CLI runs.
|
||
- `LOCAL_LLM_PATHS` – optional `os.pathsep`-separated list of extra directories to scan for `.gguf` weights.
|
||
|
||
Templates can be customized by editing `.core/templates/test-block.md`.
|
||
|
||
### Local Engine provider
|
||
|
||
- Place `.gguf` (llama.cpp) weights inside `models/` or add extra directories via the GUI settings dialog / `LOCAL_LLM_PATHS` environment variable.
|
||
- Paths include common defaults (for example `~/.lmstudio`, `~/.lmstudio/models`, or `~/Library/Application Support/lm-studio/models` on macOS) so LM Studio downloads are discovered automatically.
|
||
- The engine loader automatically selects the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA GPUs, ROCm on AMD GPUs, or a CPU fallback via `llama-cpp-python`.
|
||
- Settings are saved in `.core/user_settings.json`, keeping CLI and GUI runs in sync.
|
||
|
||
## Extending Providers
|
||
|
||
Provider adapters live under `.core/providers/` and are registered in `.core/providers/__init__.py`. Each provider implements `list_models` and `run_prompt` to integrate with the runner and GUI, while hardware-specific runtimes reside under `.core/.engine/`.
|