# LM-Gambit v2.0.0 — Automated LLM Diagnostic Suite LM-Gambit benchmarks local or remote Large Language Models against a suite of prompts you author yourself. It runs each question one at a time, records throughput and latency for every answer, and writes a markdown report you can grade. Version 2.0.0 replaces the Tkinter GUI with a React web interface served by a FastAPI backend. The diagnostic engine under `.core/` is unchanged — the CLI, providers, hardware runtimes and report format all behave exactly as before. --- ## Install **Requirements** - Python 3.10+ (3.11 recommended) - Node.js 20+ and npm (only to build the interface) - macOS, Linux or Windows — Apple Silicon, NVIDIA, AMD or CPU-only ```bash python -m pip install -r requirements.txt cd web && npm install && npm run build && cd .. ``` ## Run ```bash python app.py ``` That serves the API and the compiled interface from one origin and opens `http://localhost:8765` in your browser. | Flag | Purpose | |---|---| | `--port ` | Serve on a specific port (default 8765; the next free port is used if taken) | | `--host ` | Bind a different interface (default `127.0.0.1`) | | `--no-browser` | Do not open a browser window | | `--reload` | Auto-reload on Python changes | --- ## The interface | View | What it does | |---|---| | **Run** | Pick a provider, model and temperature; choose all or a subset of questions; watch results stream in per question with live throughput, token counts and TTFT. Cancel between questions — the partial report is still finalized. | | **Testing Suites** | Browse every suite. Built-ins are read-only; duplicate one to get an editable custom copy, then add, reorder and delete questions in it. | | **Reports** | Every saved report, with a per-question throughput chart, a metrics table and the full rendered markdown. | | **Playground** | Send a single prompt without touching the suite or writing a report. | | **Settings** | Default provider and temperature, model search paths, engine and folder information, and which plugins loaded. | A run keeps streaming while you browse other views, and reattaches if you reload the page mid-run. --- ## Writing questions Questions are grouped into named **suites**, in two tiers: ``` .core/suites// built-in, ships with the app, READ-ONLY at runtime suite.json {name, description, order} test1.txt ... tests// custom, full CRUD, yours ``` Five built-in suites ship enabled — `language`, `reasoning-logic`, `math-code`, `context-knowledge` and `safety` — 26 questions in total. They cannot be renamed, deleted or edited: the API returns 409, not just a disabled button. That keeps them a stable baseline for comparing models over time. **Duplicate** one to get an editable copy. One `.txt` file is one prompt. The **entire file** is sent to the model, and its **first non-empty line** doubles as the title in reports — so lead with the task: ``` Analyze the sentiment of the following customer review. Review: "I ordered the noise-cancelling headphones after ..." ``` Saving a custom suite rewrites its folder as `test1.txt … testN.txt` in the order shown, so the web interface and `auto-test.py` always agree. ### Question IDs `filename` is unique only *within* a suite — every suite has a `test1.txt`. A question is identified by its qualified **`/`** ID, and nothing anywhere keys off the bare filename. This matters more than it looks: graders derive their whole rubric from the prompt text, so handing one the wrong question's prompt produces a confident, plausible, entirely wrong score with no error to notice. ### Running a subset ```bash python auto-test.py --suites # list what is available python auto-test.py -m -t math-code # one suite python auto-test.py -m -t language,safety # several python auto-test.py -m # every BUILT-IN suite ``` A bare run covers the built-ins only, never custom suites — so it means the same thing on every machine rather than depending on local scratch work. The Run page offers the same choice, expandable to individual questions within each suite. --- ## Plugins > **Full reference lives in the app at `/docs`** — hooks, the UI contribution > vocabulary, worked examples, and a live view of what is installed. The > summary below is enough to get started. A plugin can grade answers, add sections to reports, expose HTTP endpoints, and contribute interface — nav entries, whole pages, and panels on the built-in views — without shipping a line of JavaScript or rebuilding the frontend. Two ship enabled: | Plugin | What it does | | --- | --- | | `response_checks` | Grades any answer against the constraints its own prompt states | | `code_lint` | Static analysis of code blocks in answers. Never executes them | Drop a `.py` file into `plugins/` and it loads at startup. Start from the skeleton, which documents every hook: ```bash cp plugins/_skeleton.py plugins/my_plugin.py ``` Files beginning with `_` are ignored, so the skeleton itself never runs. A plugin directory with an `__init__.py` works too, if you want to split one across files. ### Hooks Every hook is optional — define only what you need. | Hook | When | Purpose | |---|---|---| | `grade(test)` | after each answer | Score it. Return `0.0`–`1.0`, `True`/`False`, a `Grade`, or `None` to abstain | | `on_run_start(run)` | before the first question | Set up, announce, start a timer | | `on_test_complete(test)` | after each question | Log, stream elsewhere, react to failures | | `on_run_complete(run)` | when the run ends | Notify, export, archive — fires on cancel and failure too | | `report_sections(run)` | after the report is written | Return extra markdown to append | | `register_routes(router)` | at server start | Add endpoints under `/api/plugins/` | | `ui_contributions()` | when the UI loads | Nav entries, pages and panels — see `/docs` | | `register()` | once, at load | One-time setup; raising here disables the plugin | Plugins import only from `plugin_api`, which exposes `TestRecord`, `RunRecord`, `Grade` and `GradeEntry` as plain frozen dataclasses. They never touch engine or server internals. ### A minimal grader ```python # plugins/actor_check.py from plugin_api import Grade NAME = "Actor check" DESCRIPTION = "Verifies concurrency questions actually use an actor." def grade(test): if "thread-safe" not in test.prompt.lower(): return None # abstain — not my kind of question used_actor = "actor " in (test.response or "") return Grade( score=1.0 if used_actor else 0.0, label="actor" if used_actor else "no actor", ) ``` ### Contributing UI Plugins describe interface as data and the app renders it, so installing one never requires rebuilding the bundle: ```python from plugin_api import Action, NavItem, Page, Panel, StatRow, Table def ui_contributions(): base = "/api/plugins/my_plugin" return [ NavItem(label="My tool", path="/tool", icon="wrench", order=50), Page(path="/tool", title="My tool", blocks=[ StatRow(source=f"{base}/stats"), Panel(title="Results", blocks=[ Table(source=f"{base}/rows"), Action(label="Clear", post=f"{base}/clear", style="ghost"), ]), ]), ] ``` Blocks are `StatRow`, `Table`, `Markdown`, `Action` and `Panel`; surfaces are `NavItem`, `Page` and `SlotPanel`. Any block may take inline data or a `source` URL it fetches live. Paths are namespaced under `/x//`, so plugins cannot collide with each other or shadow a built-in route. Full vocabulary at `/docs`. ### The bundled graders `plugins/code_lint.py` statically analyses code blocks in answers — syntax, compilation, the signature the prompt demanded, stub bodies, undefined names, unused imports and more. It **never executes model output**; every check is parse-level, so a broken or hostile snippet cannot affect the grading machine. It abstains on answers with no code. `plugins/response_checks.py` ships enabled. It grades **any** answer — prose, maths, JSON, translation, code — by deriving a rubric from the prompt itself rather than from a hard-coded per-question key. Nothing in it is domain-specific. It reads the prompt for constraints the answer can be measured against, then scores each as a fraction rather than a pass/fail: | Check | Fires when the prompt… | | --- | --- | | `structure/json` | asks for valid JSON — the answer must parse | | `structure/table` | asks for a table | | `structure/code` | names a language or a `def`, or forbids a code fence | | `coverage/enumerated` | lists numbered deliverables or `PART`/`STAGE` blocks | | `coverage/keyterms` | backticks a symbol or quotes a key/heading to emit | | `constraint/forbidden` | bans a symbol ("do not use `INIntent`") | | `constraint/length` | states one unambiguous word budget | | `behaviour/abstention` | unconditionally requires a refusal or an "I don't know" | | `quality/degeneration` | always — catches looping, truncation, empty output | | `quality/substance` | always — catches one-line non-answers | Only applicable checks count toward the score, so a prompt that never mentions JSON is never scored on JSON. Three deliberate conservatisms keep it from punishing correct work: - A word budget is enforced **only when it unambiguously covers the whole answer**. A prompt asking for two summaries of different lengths skips the check rather than failing it. - A **conditional** demand ("if the constraints are contradictory, say so") is never scored — whether the condition holds is exactly what the grader cannot determine. Only unconditional demands ("do not guess") count. - **Multi-word quoted spans are ignored** as requirements. They are nearly always material being discussed — a line of the source text, or a manipulative phrase the answer is asked to quote back — not something to reproduce. The report gains a `Compliance by check` table showing which dimensions the model struggled with across the whole run, and a list of questions with unmet constraints. A low score means the answer did not do what it was told, which is not the same as being wrong — read the answer before treating it as a failure. ### How grading behaves - **Abstaining is free.** Returning `None` excludes the question from that grader entirely; it never counts as a zero. - **A question's score is the mean** of every grader that scored it. The run's score is the mean of those per-question scores. - **Failed questions are never graded** — there is no answer to judge. Use `on_test_complete` if you want to see failures. - **Grades replace the report's "grade this by hand" placeholder** with a table of scores, a letter grade, and each grader's notes. With no graders installed, the report is unchanged from v1. - Scores appear live in the run feed and in the Reports table. ### Failure isolation A plugin that raises is logged with its slug and skipped for that call only — it stays loaded and its other hooks keep firing. A plugin that fails to import is listed in Settings → Plugins with the error, and everything else still runs. A plugin can never fail a run or stop the server. ### Reloading **Settings → Plugins → Reload** re-scans the directory, picking up new graders and lifecycle hooks without a restart. Plugins that add HTTP routes need a full restart, since routes are bound when the server starts. Set `ENABLED = False` in a plugin to keep the file but stop loading it. --- ## CLI The CLI does not require the frontend to be built. ```bash python auto-test.py -p "Local Engine" -m gemma-4-E2B-it-Q8_0 -t math-code ``` | Flag | Purpose | |---|---| | `-h, --help` | Usage instructions | | `-p ` | Provider to use (e.g. `"Local Engine"`, `"LM Studio"`) | | `-m ` | Model by id or filename | | `-l, --list` | List providers, or models when combined with `-p` | | `-s, --suites` | List available suites | | `-t, --test ` | Suites to run — repeatable or comma-separated. Omit for every built-in | Reports are written to `results/automated_report_.md`. --- ## Project structure ``` .core/ Diagnostic engine (unchanged) providers/ Provider adapters (LM Studio, Local Engine) .engine/ Hardware runtimes (MLX, CUDA, ROCm, CPU) runner.py Orchestrates a run and its report reporting.py Markdown report generation prompts.py Facade over suites.py (kept for the runner) suites.py Named suites: discovery, loading, CRUD templates/ test-block.md — the per-question report template server/ FastAPI backend wrapping the engine core_bridge.py Import shim for the hidden .core package api.py REST endpoints run_manager.py Background runs + server-sent-event streaming suite.py Server layer over the core suite module plugins.py Bridge between server internals and the plugin API plugins/ Drop-in plugins — _skeleton.py is the starter template plugin_api.py Stable types plugins import plugin_system.py Plugin discovery and hook dispatch web/ React + TypeScript interface (Vite) models/ Drop .gguf or MLX weights here (auto-discovered) .core/suites// Built-in suites, read-only tests// Custom suites, one prompt per .txt results/ Generated markdown reports app.py Web entrypoint auto-test.py CLI entrypoint ``` --- ## How a run works 1. **Provider selection** — providers are registered in `.core/providers/`, each implementing `list_models()` and `run_prompt()`. The Local Engine picks the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA, ROCm on AMD, or a `llama-cpp-python` CPU fallback. 2. **Prompt loading** — the selected suites, each in natural filename order, tagged with a qualified `/` ID. 3. **Execution** — one prompt at a time against the selected model. The backend streams each result to the browser as it lands, so nothing is buffered until the end. 4. **Reporting** — each result is rendered through `.core/templates/test-block.md`, with a performance summary written at the top once the run finishes. --- ## API The backend is a normal REST API — interactive docs at `http://localhost:8765/api/docs`. | Endpoint | Purpose | |---|---| | `GET /api/providers` · `GET /api/providers/{name}/models` | Discovery | | `GET /api/suites` · `GET /api/suites/{slug}` | List suites, read one with its questions | | `POST /api/suites` · `PUT /api/suites/{slug}` · `DELETE` | Create, rename, remove a custom suite (409 on built-ins) | | `PUT /api/suites/{slug}/tests` | Replace a custom suite's questions (409 on built-ins) | | `POST /api/suites/{slug}/duplicate` | Clone any suite into an editable custom one | | `POST /api/runs` · `GET /api/runs/{id}/events` · `POST /api/runs/{id}/cancel` | Start, stream, cancel | | `GET /api/reports` · `GET /api/reports/{name}` | Saved reports | | `POST /api/playground` | One-off prompt | | `GET /api/settings` · `PUT /api/settings` | Preferences | | `GET /api/plugins` · `POST /api/plugins/reload` | Installed plugins | | `/api/plugins//…` | Whatever a plugin's `register_routes` defines | Only one run executes at a time — the local engine loads weights into memory, so overlapping runs would compete for RAM and produce meaningless throughput numbers. --- ## Configuration | Variable | Purpose | |---|---| | `LM_STUDIO_BASE_URL` | LM Studio endpoint (default `http://localhost:1234`) | | `AUTO_TEST_TEMPERATURE` | Default sampling temperature | | `AUTO_TEST_PROVIDER` | Default provider for CLI runs | | `LOCAL_LLM_PATHS` | `os.pathsep`-separated extra directories to scan for `.gguf` weights | Settings changed in the interface persist to `.core/user_settings.json` and are shared with the CLI. Common LM Studio locations (`~/.lmstudio`, `~/.lmstudio/models`, `~/Library/Application Support/lm-studio/models`) are scanned by default. --- ## Frontend development ```bash python app.py --no-browser # terminal 1 — API on :8765 cd web && npm run dev # terminal 2 — UI on :5173 with hot reload ``` Vite proxies `/api` (including the event stream) through to the Python server, so both run from one origin. --- ## Extending **A new provider** — add a class in `.core/providers/` inheriting from `Provider`, implement `list_models()` and `run_prompt()`, and register it in `.core/providers/__init__.py`. It appears in the interface automatically. **A new engine runtime** — add `.core/.engine/./.py` defining an `EngineRuntime` class that inherits from `BaseRuntime`. The loader picks it up when the hardware matches. --- ## Troubleshooting **"Frontend not built"** — run `cd web && npm install && npm run build`. The API and CLI work without it. **No models found** — put `.gguf` files in `models/`, add a path under Settings, or set `LOCAL_LLM_PATHS`. For LM Studio, make sure the app is running with its API enabled. **Port already in use** — `app.py` automatically tries the next 20 ports, or pass `--port`. **Engine errors** — check Settings → Engine for the detected runtime, and confirm your hardware drivers and Python dependencies are installed. --- ## License MIT License. See `LICENSE` for details.