Initial commit: Original Python LLMTester project

2025-10-03 01:11:39 -04:00
commit ac5204e93a
54 changed files with 11911 additions and 0 deletions
@@ -0,0 +1,77 @@
+# LLM Automated Test Suite
+
+This project automates a battery of diagnostic prompts against local or remote language models via a modular core runner and an optional GUI.
+
+## Prerequisites
+
+- Python 3.10 or newer (tkinter included with the standard library on macOS/Linux; install `python3-tk` on Debian-based systems if needed).
+- Install dependencies:
+
+```bash
+python -m pip install -r requirements.txt
+```
+
+## Project Layout
+
+- `.core/` – shared logic modules, templates, engines, and transient workspace used by both CLI and GUI.
+  - `config.py` – path/setting helpers.
+  - `prompts.py` – prompt discovery from `tests/`.
+  - `providers/` – provider adapters (LM Studio and the native Local Engine runtime).
+  - `reporting.py` – markdown templating, code-fence hygiene, report summaries.
+  - `runner.py` – orchestrates full test runs with progress callbacks.
+  - `engine_loader.py` – detects the host platform/GPU and loads the appropriate runtime.
+  - `.engine/.<architecture>/<version>.py` – runtime implementations per hardware family (Apple Silicon ↦ MLX, CUDA ↦ llama.cpp, ROCm ↦ llama.cpp, CPU fallback).
+  - `.temp/` – ephemeral staging for rendered blocks.
+  - `templates/test-block.md` – markdown template for each test result.
+- `models/` – drop-in directory for local `.gguf` (and MLX-compatible) weights discovered by the Local Engine provider.
+- `tests/` – one prompt per `.txt` file (editable via the GUI or any editor).
+- `results/` – generated markdown reports.
+- `auto-test.py` – CLI entrypoint that runs the default provider/model.
+- `index.py` – Tkinter GUI for provider/model selection, running tests, and opening prompts/results.
+
+## Running the CLI
+
+```bash
+python auto-test.py
+```
+
+The CLI selects the first available model for the default provider (LM Studio by default) and streams progress to stdout. Reports are saved in `results/automated_report_<model>.md`.
+
+To switch to the Local Engine runtime, set `AUTO_TEST_PROVIDER="Local Engine"` (or choose it in the GUI) and ensure your models are available in the `models/` directory or any path listed in `LOCAL_LLM_PATHS`.
+
+## Using the GUI
+
+```bash
+python index.py
+```
+
+GUI features:
+
+- Provider dropdown with automatic model discovery.
+- Model dropdown populated per provider.
+- Adjustable sampling temperature.
+- Buttons to run tests, refresh models, open the `tests/` folder, open the `results/` folder, and open the latest report.
+- Live log of test progress and completion status.
+- Settings menu → “Configure Model Paths…” to manage additional folders scanned by the Local Engine provider (persisted in `.core/user_settings.json`).
+
+## Configuration
+
+Environment variables:
+
+- `LM_STUDIO_BASE_URL` – override the default `http://localhost:1234` endpoint for the LM Studio provider.
+- `AUTO_TEST_TEMPERATURE` – default sampling temperature for test runs.
+- `AUTO_TEST_PROVIDER` – default provider name for CLI runs.
+- `LOCAL_LLM_PATHS` – optional `os.pathsep`-separated list of extra directories to scan for `.gguf` weights.
+
+Templates can be customized by editing `.core/templates/test-block.md`.
+
+### Local Engine provider
+
+- Place `.gguf` (llama.cpp) weights inside `models/` or add extra directories via the GUI settings dialog / `LOCAL_LLM_PATHS` environment variable.
+- Paths include common defaults (for example `~/.lmstudio`, `~/.lmstudio/models`, or `~/Library/Application Support/lm-studio/models` on macOS) so LM Studio downloads are discovered automatically.
+- The engine loader automatically selects the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA GPUs, ROCm on AMD GPUs, or a CPU fallback via `llama-cpp-python`.
+- Settings are saved in `.core/user_settings.json`, keeping CLI and GUI runs in sync.
+
+## Extending Providers
+
+Provider adapters live under `.core/providers/` and are registered in `.core/providers/__init__.py`. Each provider implements `list_models` and `run_prompt` to integrate with the runner and GUI, while hardware-specific runtimes reside under `.core/.engine/`.