christopher/LM-Gambit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Settings and Suite pages with comprehensive settings management and question suite builder

Browse Source 
- Implemented SettingsPage for configuring default provider, temperature, and model paths.
- Added SuitePage for managing a suite of questions with features to add, edit, duplicate, and delete questions.
- Introduced TypeScript configuration files for app and node environments.
- Set up Vite configuration for development server with API proxying to backend.
This commit is contained in:
Christopher Clendening
2026-07-27 17:41:08 -04:00
parent ce0e022f11
commit adf61ae1a0
50 changed files with 10952 additions and 164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+237 -131
View File
@@ -1,191 +1,297 @@
# LM-Gambit v2.0.0 — Automated LLM Diagnostic Suite
LM-Gambit benchmarks local or remote Large Language Models against a suite of prompts you
author yourself. It runs each question one at a time, records throughput and latency for
every answer, and writes a markdown report you can grade.
# LM-Gambit v1.0.0: Automated LLM Diagnostic Suite
**Version 1.0.0**
LM-Gambit is a modular, extensible framework for benchmarking and analyzing local or remote Large Language Models (LLMs). It supports both a command-line interface (CLI) and a Tkinter-based GUI, enabling rapid, repeatable evaluation of LLMs using customizable prompt suites.
Version 2.0.0 replaces the Tkinter GUI with a React web interface served by a FastAPI
backend. The diagnostic engine under `.core/` is unchanged — the CLI, providers, hardware
runtimes and report format all behave exactly as before.
---
## Features
## Install
- **Automated Testing:** Run a suite of prompt-based diagnostics against any supported LLM provider.
- **Provider System:** Out-of-the-box support for LM Studio (API) and a native Local Engine (hardware-optimized, supports Apple Silicon, CUDA, ROCm, CPU).
- **Extensible:** Add new providers or engine runtimes with minimal code changes.
- **GUI & CLI:** Choose between a full-featured GUI or a fast CLI for scripting and automation.
- **Customizable Reports:** Generates detailed Markdown reports with performance metrics and qualitative analysis sections.
- **Prompt Management:** Prompts are simple `.txt` files, editable in the GUI or any text editor.
**Requirements**
---
## Installation
**Requirements:**
- Python 3.10+
- macOS, Linux, or Windows (Apple Silicon, NVIDIA, AMD, or CPU-only supported)
**Install dependencies:**
- Python 3.10+ (3.11 recommended)
- Node.js 20+ and npm (only to build the interface)
- macOS, Linux or Windows — Apple Silicon, NVIDIA, AMD or CPU-only
```bash
python -m pip install -r requirements.txt
cd web && npm install && npm run build && cd ..
```
**(Optional)** For Apple Silicon (MLX), ensure `mlx-lm` is installed. For CUDA/ROCm, ensure your system drivers are set up.
## Run
```bash
python app.py
```
That serves the API and the compiled interface from one origin and opens
`http://localhost:8765` in your browser.
| Flag | Purpose |
|---|---|
| `--port <n>` | Serve on a specific port (default 8765; the next free port is used if taken) |
| `--host <addr>` | Bind a different interface (default `127.0.0.1`) |
| `--no-browser` | Do not open a browser window |
| `--reload` | Auto-reload on Python changes |
---
## Quick Start
## The interface
### CLI Usage
| View | What it does |
|---|---|
| **Run** | Pick a provider, model and temperature; choose all or a subset of questions; watch results stream in per question with live throughput, token counts and TTFT. Cancel between questions — the partial report is still finalized. |
| **Suite** | The question editor. One text box per question, with add, duplicate, reorder and delete. |
| **Reports** | Every saved report, with a per-question throughput chart, a metrics table and the full rendered markdown. |
| **Playground** | Send a single prompt without touching the suite or writing a report. |
| **Settings** | Default provider and temperature, model search paths, engine and folder information, and which plugins loaded. |
Run all tests with the default provider/model:
A run keeps streaming while you browse other views, and reattaches if you reload the page
mid-run.
```bash
python auto-test.py
---
## Writing questions
Questions live in `tests/` as one `.txt` file per prompt. The **entire file** is sent to the
model, and its **first non-empty line** doubles as the title in reports — so lead with the
task:
```
Write a Swift function that solves the FizzBuzz problem.
The function must have the exact signature:
`func generateFizzBuzz(upTo max: Int) -> [String]`
```
**Options:**
- `-h, --help` — Show usage instructions
- `-p <provider>` — Select provider (e.g., "Local Engine", "LM Studio")
- `-m <model>` — Select model by ID or filename
- `-l, --list` — List available providers or models
- `-t <test>` — (Reserved) Specify a test suite
Saving from the Suite Builder rewrites the folder as `test1.txt … testN.txt` in the order
shown, so the web interface and `auto-test.py` always run the same suite in the same order.
You can still edit the `.txt` files by hand.
Example:
---
## Plugins
Drop a `.py` file into `plugins/` and it loads at startup. Start from the
skeleton, which documents every hook:
```bash
cp plugins/_skeleton.py plugins/my_plugin.py
```
Files beginning with `_` are ignored, so the skeleton itself never runs. A
plugin directory with an `__init__.py` works too, if you want to split one
across files.
### Hooks
Every hook is optional — define only what you need.
| Hook | When | Purpose |
|---|---|---|
| `grade(test)` | after each answer | Score it. Return `0.0`–`1.0`, `True`/`False`, a `Grade`, or `None` to abstain |
| `on_run_start(run)` | before the first question | Set up, announce, start a timer |
| `on_test_complete(test)` | after each question | Log, stream elsewhere, react to failures |
| `on_run_complete(run)` | when the run ends | Notify, export, archive — fires on cancel and failure too |
| `report_sections(run)` | after the report is written | Return extra markdown to append |
| `register_routes(router)` | at server start | Add endpoints under `/api/plugins/<slug>` |
| `register()` | once, at load | One-time setup; raising here disables the plugin |
Plugins import only from `plugin_api`, which exposes `TestRecord`, `RunRecord`,
`Grade` and `GradeEntry` as plain frozen dataclasses. They never touch engine or
server internals.
### A minimal grader
```python
# plugins/actor_check.py
from plugin_api import Grade
NAME = "Actor check"
DESCRIPTION = "Verifies concurrency questions actually use an actor."
def grade(test):
if "thread-safe" not in test.prompt.lower():
return None # abstain — not my kind of question
used_actor = "actor " in (test.response or "")
return Grade(
score=1.0 if used_actor else 0.0,
label="actor" if used_actor else "no actor",
)
```
### How grading behaves
- **Abstaining is free.** Returning `None` excludes the question from that
grader entirely; it never counts as a zero.
- **A question's score is the mean** of every grader that scored it. The run's
score is the mean of those per-question scores.
- **Failed questions are never graded** — there is no answer to judge. Use
`on_test_complete` if you want to see failures.
- **Grades replace the report's "grade this by hand" placeholder** with a table
of scores, a letter grade, and each grader's notes. With no graders installed,
the report is unchanged from v1.
- Scores appear live in the run feed and in the Reports table.
### Failure isolation
A plugin that raises is logged with its slug and skipped for that call only —
it stays loaded and its other hooks keep firing. A plugin that fails to import
is listed in Settings → Plugins with the error, and everything else still runs.
A plugin can never fail a run or stop the server.
### Reloading
**Settings → Plugins → Reload** re-scans the directory, picking up new graders
and lifecycle hooks without a restart. Plugins that add HTTP routes need a full
restart, since routes are bound when the server starts.
Set `ENABLED = False` in a plugin to keep the file but stop loading it.
---
## CLI
The CLI is unchanged and does not require the frontend to be built.
```bash
python auto-test.py -p "Local Engine" -m Qwen3-Coder-30B.gguf
```
Reports are saved to `results/automated_report_<model>.md`.
| Flag | Purpose |
|---|---|
| `-h, --help` | Usage instructions |
| `-p <provider>` | Provider to use (e.g. `"Local Engine"`, `"LM Studio"`) |
| `-m <model>` | Model by id or filename |
| `-l, --list` | List providers, or models when combined with `-p` |
### GUI Usage
Reports are written to `results/automated_report_<model>.md`.
```bash
python index.py
---
## Project structure
```
.core/ Diagnostic engine (unchanged)
providers/ Provider adapters (LM Studio, Local Engine)
.engine/ Hardware runtimes (MLX, CUDA, ROCm, CPU)
runner.py Orchestrates a run and its report
reporting.py Markdown report generation
prompts.py Loads prompts from tests/
templates/ test-block.md — the per-question report template
server/ FastAPI backend wrapping the engine
core_bridge.py Import shim for the hidden .core package
api.py REST endpoints
run_manager.py Background runs + server-sent-event streaming
suite.py Reads and writes tests/
plugins.py Bridge between server internals and the plugin API
plugins/ Drop-in plugins — _skeleton.py is the starter template
plugin_api.py Stable types plugins import
plugin_system.py Plugin discovery and hook dispatch
web/ React + TypeScript interface (Vite)
models/ Drop .gguf or MLX weights here (auto-discovered)
tests/ One prompt per .txt file
results/ Generated markdown reports
app.py Web entrypoint
auto-test.py CLI entrypoint
```
**GUI Features:**
- Provider/model dropdowns with auto-discovery
- Adjustable sampling temperature
- Run, refresh, open prompts/results, and view latest report
- Live log of test progress
- Settings dialog for managing model search paths
---
## How a run works
1. **Provider selection** — providers are registered in `.core/providers/`, each implementing
`list_models()` and `run_prompt()`. The Local Engine picks the best runtime for your
hardware: MLX on Apple Silicon, CUDA on NVIDIA, ROCm on AMD, or a `llama-cpp-python` CPU
fallback.
2. **Prompt loading** — every `.txt` in `tests/`, in natural filename order.
3. **Execution** — one prompt at a time against the selected model. The backend streams each
result to the browser as it lands, so nothing is buffered until the end.
4. **Reporting** — each result is rendered through `.core/templates/test-block.md`, with a
performance summary written at the top once the run finishes.
---
## Project Structure
## API
- `.core/` — Core logic, providers, engine loader, reporting, templates
- `providers/` — Provider adapters (LM Studio, Local Engine, etc.)
- `.engine/` — Hardware-specific runtime implementations (MLX, CUDA, ROCm, CPU)
- `runner.py` — Orchestrates test runs, progress, and reporting
- `reporting.py` — Markdown report generation, code-fence hygiene
- `prompts.py` — Loads and parses prompt files from `tests/`
- `templates/test-block.md` — Customizable template for each test result
- `models/` — Drop-in directory for `.gguf` or MLX-compatible weights (auto-discovered)
- `tests/` — One prompt per `.txt` file (edit or add your own)
- `results/` — Markdown reports generated after each run
- `auto-test.py` — CLI entrypoint
- `index.py` — Tkinter GUI entrypoint
The backend is a normal REST API — interactive docs at `http://localhost:8765/api/docs`.
| Endpoint | Purpose |
|---|---|
| `GET /api/providers` · `GET /api/providers/{name}/models` | Discovery |
| `GET /api/tests` · `PUT /api/tests` | Read and replace the suite |
| `POST /api/runs` · `GET /api/runs/{id}/events` · `POST /api/runs/{id}/cancel` | Start, stream, cancel |
| `GET /api/reports` · `GET /api/reports/{name}` | Saved reports |
| `POST /api/playground` | One-off prompt |
| `GET /api/settings` · `PUT /api/settings` | Preferences |
| `GET /api/plugins` · `POST /api/plugins/reload` | Installed plugins |
| `/api/plugins/<slug>/…` | Whatever a plugin's `register_routes` defines |
Only one run executes at a time — the local engine loads weights into memory, so overlapping
runs would compete for RAM and produce meaningless throughput numbers.
---
## How the Core Engine Works
## Configuration
1. **Provider Selection:**
- Providers (e.g., LM Studio, Local Engine) are registered in `.core/providers/`.
- Each provider implements `list_models()` and `run_prompt()`.
- The Local Engine auto-selects the best runtime for your hardware (MLX, CUDA, ROCm, or CPU fallback).
| Variable | Purpose |
|---|---|
| `LM_STUDIO_BASE_URL` | LM Studio endpoint (default `http://localhost:1234`) |
| `AUTO_TEST_TEMPERATURE` | Default sampling temperature |
| `AUTO_TEST_PROVIDER` | Default provider for CLI runs |
| `LOCAL_LLM_PATHS` | `os.pathsep`-separated extra directories to scan for `.gguf` weights |
2. **Prompt Loading:**
- Prompts are loaded from the `tests/` directory (one `.txt` file per test).
- Each prompt file's first non-empty line is used as the test title.
3. **Test Execution:**
- For each prompt, the selected provider/model is used to generate a response.
- Results are streamed to the CLI or GUI with live progress updates.
4. **Reporting:**
- Each test result is rendered using a Markdown template (`.core/templates/test-block.md`).
- Performance metrics (tokens/sec, time to first token, etc.) are included.
- A summary and qualitative analysis section are generated at the top of the report.
Settings changed in the interface persist to `.core/user_settings.json` and are shared with
the CLI. Common LM Studio locations (`~/.lmstudio`, `~/.lmstudio/models`,
`~/Library/Application Support/lm-studio/models`) are scanned by default.
---
## Configuration & Environment Variables
## Frontend development
You can customize behavior via environment variables:
```bash
python app.py --no-browser # terminal 1 — API on :8765
cd web && npm run dev # terminal 2 — UI on :5173 with hot reload
```
| Variable | Purpose |
|-------------------------|--------------------------------------------------------------|
| `LM_STUDIO_BASE_URL` | Override LM Studio API endpoint (default: http://localhost:1234) |
| `AUTO_TEST_TEMPERATURE` | Default sampling temperature for test runs |
| `AUTO_TEST_PROVIDER` | Default provider for CLI runs |
| `LOCAL_LLM_PATHS` | `:`-separated list of extra directories for `.gguf` models |
Model search paths can also be managed via the GUI settings dialog (persisted in `.core/user_settings.json`).
Vite proxies `/api` (including the event stream) through to the Python server, so both run
from one origin.
---
## Extending
## Extending LM-Gambit
**A new provider** — add a class in `.core/providers/` inheriting from `Provider`, implement
`list_models()` and `run_prompt()`, and register it in `.core/providers/__init__.py`. It
appears in the interface automatically.
**Add a new provider:**
1. Create a new class in `.core/providers/` inheriting from `Provider` (see `base.py`).
2. Implement `list_models()` and `run_prompt()`.
3. Register your provider in `.core/providers/__init__.py`.
**Add a new engine runtime:**
1. Add a new Python file under `.core/.engine/.<architecture>/<version>.py`.
2. Implement an `EngineRuntime` class inheriting from `BaseRuntime`.
3. The engine loader will auto-detect and use your runtime if the hardware matches.
---
## Customizing Prompts & Reports in LM-Gambit
- Add/edit prompt `.txt` files in `tests/` (first non-empty line is the title).
- Edit `.core/templates/test-block.md` to change the report format.
**A new engine runtime** — add `.core/.engine/.<architecture>/<version>.py` defining an
`EngineRuntime` class that inherits from `BaseRuntime`. The loader picks it up when the
hardware matches.
---
## Troubleshooting
- **No models found?**
- Ensure your `.gguf` files are in `models/` or listed in `LOCAL_LLM_PATHS`.
- For LM Studio, ensure the app is running and the API is enabled.
- **Engine errors?**
- Check your hardware drivers and Python dependencies.
- **GUI not launching?**
- Ensure `tkinter` is installed and available in your Python environment.
**"Frontend not built"** — run `cd web && npm install && npm run build`. The API and CLI work
without it.
**No models found** — put `.gguf` files in `models/`, add a path under Settings, or set
`LOCAL_LLM_PATHS`. For LM Studio, make sure the app is running with its API enabled.
**Port already in use** — `app.py` automatically tries the next 20 ports, or pass `--port`.
**Engine errors** — check Settings → Engine for the detected runtime, and confirm your
hardware drivers and Python dependencies are installed.
---
## License
MIT License. See `LICENSE` for details.
- `LM_STUDIO_BASE_URL` – override the default `http://localhost:1234` endpoint for the LM Studio provider.
- `AUTO_TEST_TEMPERATURE` – default sampling temperature for test runs.
- `AUTO_TEST_PROVIDER` – default provider name for CLI runs.
- `LOCAL_LLM_PATHS` – optional `os.pathsep`-separated list of extra directories to scan for `.gguf` weights.
Templates can be customized by editing `.core/templates/test-block.md`.
### Local Engine provider
- Place `.gguf` (llama.cpp) weights inside `models/` or add extra directories via the GUI settings dialog / `LOCAL_LLM_PATHS` environment variable.
- Paths include common defaults (for example `~/.lmstudio`, `~/.lmstudio/models`, or `~/Library/Application Support/lm-studio/models` on macOS) so LM Studio downloads are discovered automatically.
- The engine loader automatically selects the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA GPUs, ROCm on AMD GPUs, or a CPU fallback via `llama-cpp-python`.
- Settings are saved in `.core/user_settings.json`, keeping CLI and GUI runs in sync.
## Extending Providers in LM-Gambit
Provider adapters live under `.core/providers/` and are registered in `.core/providers/__init__.py`. Each provider implements `list_models` and `run_prompt` to integrate with the runner and GUI, while hardware-specific runtimes reside under `.core/.engine/`.