christopher/LM-Gambit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Release v1.0.0: update README and GUI title bar for version 1.0.0

Browse Source
This commit is contained in:
christopher
2025-10-03 05:37:15 -04:00
parent 05b069d241
commit b78cd43464
2 changed files with 152 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+149 -35
View File
@@ -1,62 +1,175 @@
# LLM Automated Test Suite
This project automates a battery of diagnostic prompts against local or remote language models via a modular core runner and an optional GUI.
## Prerequisites
- Python 3.10 or newer (tkinter included with the standard library on macOS/Linux; install `python3-tk` on Debian-based systems if needed).
- Install dependencies:
# LM-Gambit v1.0.0: Automated LLM Diagnostic Suite
**Version 1.0.0**
LM-Gambit is a modular, extensible framework for benchmarking and analyzing local or remote Large Language Models (LLMs). It supports both a command-line interface (CLI) and a Tkinter-based GUI, enabling rapid, repeatable evaluation of LLMs using customizable prompt suites.
---
## Features
- **Automated Testing:** Run a suite of prompt-based diagnostics against any supported LLM provider.
- **Provider System:** Out-of-the-box support for LM Studio (API) and a native Local Engine (hardware-optimized, supports Apple Silicon, CUDA, ROCm, CPU).
- **Extensible:** Add new providers or engine runtimes with minimal code changes.
- **GUI & CLI:** Choose between a full-featured GUI or a fast CLI for scripting and automation.
- **Customizable Reports:** Generates detailed Markdown reports with performance metrics and qualitative analysis sections.
- **Prompt Management:** Prompts are simple `.txt` files, editable in the GUI or any text editor.
---
## Installation
**Requirements:**
- Python 3.10+
- macOS, Linux, or Windows (Apple Silicon, NVIDIA, AMD, or CPU-only supported)
**Install dependencies:**
```bash
python -m pip install -r requirements.txt
```
## Project Layout
**(Optional)** For Apple Silicon (MLX), ensure `mlx-lm` is installed. For CUDA/ROCm, ensure your system drivers are set up.
- `.core/` – shared logic modules, templates, engines, and transient workspace used by both CLI and GUI.
- `config.py` – path/setting helpers.
- `prompts.py` – prompt discovery from `tests/`.
- `providers/` – provider adapters (LM Studio and the native Local Engine runtime).
- `reporting.py` – markdown templating, code-fence hygiene, report summaries.
- `runner.py` – orchestrates full test runs with progress callbacks.
- `engine_loader.py` – detects the host platform/GPU and loads the appropriate runtime.
- `.engine/.<architecture>/<version>.py` – runtime implementations per hardware family (Apple Silicon ↦ MLX, CUDA ↦ llama.cpp, ROCm ↦ llama.cpp, CPU fallback).
- `.temp/` – ephemeral staging for rendered blocks.
- `templates/test-block.md` – markdown template for each test result.
- `models/` – drop-in directory for local `.gguf` (and MLX-compatible) weights discovered by the Local Engine provider.
- `tests/` – one prompt per `.txt` file (editable via the GUI or any editor).
- `results/` – generated markdown reports.
- `auto-test.py` – CLI entrypoint that runs the default provider/model.
- `index.py` – Tkinter GUI for provider/model selection, running tests, and opening prompts/results.
---
## Running the CLI
## Quick Start
### CLI Usage
Run all tests with the default provider/model:
```bash
python auto-test.py
```
The CLI selects the first available model for the default provider (LM Studio by default) and streams progress to stdout. Reports are saved in `results/automated_report_<model>.md`.
**Options:**
- `-h, --help` — Show usage instructions
- `-p <provider>` — Select provider (e.g., "Local Engine", "LM Studio")
- `-m <model>` — Select model by ID or filename
- `-l, --list` — List available providers or models
- `-t <test>` — (Reserved) Specify a test suite
To switch to the Local Engine runtime, set `AUTO_TEST_PROVIDER="Local Engine"` (or choose it in the GUI) and ensure your models are available in the `models/` directory or any path listed in `LOCAL_LLM_PATHS`.
Example:
## Using the GUI
```bash
python auto-test.py -p "Local Engine" -m Qwen3-Coder-30B.gguf
```
Reports are saved to `results/automated_report_<model>.md`.
### GUI Usage
```bash
python index.py
```
GUI features:
**GUI Features:**
- Provider/model dropdowns with auto-discovery
- Adjustable sampling temperature
- Run, refresh, open prompts/results, and view latest report
- Live log of test progress
- Settings dialog for managing model search paths
- Provider dropdown with automatic model discovery.
- Model dropdown populated per provider.
- Adjustable sampling temperature.
- Buttons to run tests, refresh models, open the `tests/` folder, open the `results/` folder, and open the latest report.
- Live log of test progress and completion status.
- Settings menu → “Configure Model Paths…” to manage additional folders scanned by the Local Engine provider (persisted in `.core/user_settings.json`).
---
## Configuration
## Project Structure
Environment variables:
- `.core/` — Core logic, providers, engine loader, reporting, templates
- `providers/` — Provider adapters (LM Studio, Local Engine, etc.)
- `.engine/` — Hardware-specific runtime implementations (MLX, CUDA, ROCm, CPU)
- `runner.py` — Orchestrates test runs, progress, and reporting
- `reporting.py` — Markdown report generation, code-fence hygiene
- `prompts.py` — Loads and parses prompt files from `tests/`
- `templates/test-block.md` — Customizable template for each test result
- `models/` — Drop-in directory for `.gguf` or MLX-compatible weights (auto-discovered)
- `tests/` — One prompt per `.txt` file (edit or add your own)
- `results/` — Markdown reports generated after each run
- `auto-test.py` — CLI entrypoint
- `index.py` — Tkinter GUI entrypoint
---
## How the Core Engine Works
1. **Provider Selection:**
- Providers (e.g., LM Studio, Local Engine) are registered in `.core/providers/`.
- Each provider implements `list_models()` and `run_prompt()`.
- The Local Engine auto-selects the best runtime for your hardware (MLX, CUDA, ROCm, or CPU fallback).
2. **Prompt Loading:**
- Prompts are loaded from the `tests/` directory (one `.txt` file per test).
- Each prompt file's first non-empty line is used as the test title.
3. **Test Execution:**
- For each prompt, the selected provider/model is used to generate a response.
- Results are streamed to the CLI or GUI with live progress updates.
4. **Reporting:**
- Each test result is rendered using a Markdown template (`.core/templates/test-block.md`).
- Performance metrics (tokens/sec, time to first token, etc.) are included.
- A summary and qualitative analysis section are generated at the top of the report.
---
## Configuration & Environment Variables
You can customize behavior via environment variables:
| Variable | Purpose |
|-------------------------|--------------------------------------------------------------|
| `LM_STUDIO_BASE_URL` | Override LM Studio API endpoint (default: http://localhost:1234) |
| `AUTO_TEST_TEMPERATURE` | Default sampling temperature for test runs |
| `AUTO_TEST_PROVIDER` | Default provider for CLI runs |
| `LOCAL_LLM_PATHS` | `:`-separated list of extra directories for `.gguf` models |
Model search paths can also be managed via the GUI settings dialog (persisted in `.core/user_settings.json`).
---
## Extending LM-Gambit
**Add a new provider:**
1. Create a new class in `.core/providers/` inheriting from `Provider` (see `base.py`).
2. Implement `list_models()` and `run_prompt()`.
3. Register your provider in `.core/providers/__init__.py`.
**Add a new engine runtime:**
1. Add a new Python file under `.core/.engine/.<architecture>/<version>.py`.
2. Implement an `EngineRuntime` class inheriting from `BaseRuntime`.
3. The engine loader will auto-detect and use your runtime if the hardware matches.
---
## Customizing Prompts & Reports in LM-Gambit
- Add/edit prompt `.txt` files in `tests/` (first non-empty line is the title).
- Edit `.core/templates/test-block.md` to change the report format.
---
## Troubleshooting
- **No models found?**
- Ensure your `.gguf` files are in `models/` or listed in `LOCAL_LLM_PATHS`.
- For LM Studio, ensure the app is running and the API is enabled.
- **Engine errors?**
- Check your hardware drivers and Python dependencies.
- **GUI not launching?**
- Ensure `tkinter` is installed and available in your Python environment.
---
## License
MIT License. See `LICENSE` for details.
- `LM_STUDIO_BASE_URL` – override the default `http://localhost:1234` endpoint for the LM Studio provider.
- `AUTO_TEST_TEMPERATURE` – default sampling temperature for test runs.
@@ -72,6 +185,7 @@ Templates can be customized by editing `.core/templates/test-block.md`.
- The engine loader automatically selects the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA GPUs, ROCm on AMD GPUs, or a CPU fallback via `llama-cpp-python`.
- Settings are saved in `.core/user_settings.json`, keeping CLI and GUI runs in sync.
## Extending Providers
## Extending Providers in LM-Gambit
Provider adapters live under `.core/providers/` and are registered in `.core/providers/__init__.py`. Each provider implements `list_models` and `run_prompt` to integrate with the runner and GUI, while hardware-specific runtimes reside under `.core/.engine/`.