LM-Gambit/README.md

LM-Gambit v2.0.0 — Automated LLM Diagnostic Suite

LM-Gambit benchmarks local or remote Large Language Models against a suite of prompts you author yourself. It runs each question one at a time, records throughput and latency for every answer, and writes a markdown report you can grade.

What 2.0.0 brings over 1.0.0

The Tkinter GUI is replaced by a React web interface served by a FastAPI backend. Beyond the interface:

• Named suites. Questions live in five read-only built-in suites rather than one flat directory, and a run can draw from several at once, or from a subset of any. Custom suites are yours to edit; built-ins stay a stable baseline for comparison.
• Three graders. Two measure form — whether an answer did what it was told — and answer_key measures whether it was actually right, for the questions with an unambiguous answer.
• A plugin framework. Plugins add nav entries, pages and panels by describing them as data, so installing one never requires rebuilding the frontend.
• Run history. Reports are named <model>__<timestamp>__<scope>__<n>q.md, so runs no longer overwrite each other and each file records what it covered.

The auto-test.py CLI still drives the same engine the web interface does.

---

Install

Requirements

• Python 3.10+ (3.11 recommended)
• Node.js 20+ and npm (only to build the interface)
• macOS, Linux or Windows — Apple Silicon, NVIDIA, AMD or CPU-only

From a release archive

Download the archive for your platform from Releases, extract it, and run the launcher inside: LM-Gambit.bat on Windows, ./LM-Gambit.sh on macOS and Linux. It creates a virtual environment and installs dependencies on first run — a few minutes, once — then starts the app. The interface is prebuilt, so Node is not needed.

Python 3.10+ still has to be on the machine. The archive is not a frozen binary, and that is deliberate: llama-cpp-python compiles its GPU backend in at install time, so anything prebuilt would lock every user to whatever hardware the build machine had. Installing on your machine lets pip pick the matching build. See GPU-ACCELERATION.md in the archive for CUDA, Metal and ROCm.

From source

python -m pip install -r requirements.txt
cd web && npm install && npm run build && cd ..

Run

python app.py

That serves the API and the compiled interface from one origin and opens http://localhost:8765 in your browser.

Flag	Purpose
--port <n>	Serve on a specific port (default 8765; the next free port is used if taken)
--host <addr>	Bind a different interface (default 127.0.0.1)
--no-browser	Do not open a browser window
--reload	Auto-reload on Python changes

---

The interface

View	What it does
Run	Pick a provider, model and temperature; choose all or a subset of questions; watch results stream in per question with live throughput, token counts and TTFT. Cancel between questions — the partial report is still finalized.
Testing Suites	Browse every suite. Built-ins are read-only; duplicate one to get an editable custom copy, then add, reorder and delete questions in it.
Reports	Every saved report, with a per-question throughput chart, a metrics table and the full rendered markdown.
Playground	Send a single prompt without touching the suite or writing a report.
Settings	Default provider and temperature, model search paths, engine and folder information, and which plugins loaded.

A run keeps streaming while you browse other views, and reattaches if you reload the page mid-run.

---

Writing questions

Questions are grouped into named suites, in two tiers:

.core/suites/<slug>/     built-in, ships with the app, READ-ONLY at runtime
    suite.json           {name, description, order}
    test1.txt ...
tests/<slug>/            custom, full CRUD, yours

Five built-in suites ship enabled — language, reasoning-logic, math-code, context-knowledge and safety — 26 questions in total. They cannot be renamed, deleted or edited: the API returns 409, not just a disabled button. That keeps them a stable baseline for comparing models over time. Duplicate one to get an editable copy.

One .txt file is one prompt. The entire file is sent to the model, and its first non-empty line doubles as the title in reports — so lead with the task:

Analyze the sentiment of the following customer review.

Review:
"I ordered the noise-cancelling headphones after ..."

Saving a custom suite rewrites its folder as test1.txt … testN.txt in the order shown, so the web interface and auto-test.py always agree.

Question IDs

filename is unique only within a suite — every suite has a test1.txt. A question is identified by its qualified <suite>/<file> ID, and nothing anywhere keys off the bare filename. This matters more than it looks: graders derive their whole rubric from the prompt text, so handing one the wrong question's prompt produces a confident, plausible, entirely wrong score with no error to notice.

Running a subset

python auto-test.py --suites                       # list what is available
python auto-test.py -m <model> -t math-code        # one suite
python auto-test.py -m <model> -t language,safety  # several
python auto-test.py -m <model>                     # every BUILT-IN suite

A bare run covers the built-ins only, never custom suites — so it means the same thing on every machine rather than depending on local scratch work. The Run page offers the same choice, expandable to individual questions within each suite.

---

Plugins

Full reference lives in the app at /docs — hooks, the UI contribution vocabulary, worked examples, and a live view of what is installed. The summary below is enough to get started.

A plugin can grade answers, add sections to reports, expose HTTP endpoints, and contribute interface — nav entries, whole pages, and panels on the built-in views — without shipping a line of JavaScript or rebuilding the frontend.

Three ship enabled:

Plugin	What it does
response_checks	Grades any answer against the constraints its own prompt states
code_lint	Static analysis of code blocks in answers. Never executes them
answer_key	Checks answers against known-correct content, where one exists

The first two measure form — whether an answer did what it was told. answer_key is the only one that asks whether it was right, and it only covers questions with an unambiguous answer (16 of 26). The rest abstain, so a high score reads as "correct where checkable, well-formed elsewhere".

Drop a .py file into plugins/ and it loads at startup. Start from the skeleton, which documents every hook:

cp plugins/_skeleton.py plugins/my_plugin.py

Files beginning with _ are ignored, so the skeleton itself never runs. A plugin directory with an __init__.py works too, if you want to split one across files.

Hooks

Every hook is optional — define only what you need.

Hook	When	Purpose
grade(test)	after each answer	Score it. Return 0.0–1.0, True/False, a Grade, or None to abstain
on_run_start(run)	before the first question	Set up, announce, start a timer
on_test_complete(test)	after each question	Log, stream elsewhere, react to failures
on_run_complete(run)	when the run ends	Notify, export, archive — fires on cancel and failure too
report_sections(run)	after the report is written	Return extra markdown to append
register_routes(router)	at server start	Add endpoints under /api/plugins/<slug>
ui_contributions()	when the UI loads	Nav entries, pages and panels — see /docs
register()	once, at load	One-time setup; raising here disables the plugin

Plugins import only from plugin_api, which exposes TestRecord, RunRecord, Grade and GradeEntry as plain frozen dataclasses. They never touch engine or server internals.

A minimal grader

# plugins/actor_check.py
from plugin_api import Grade

NAME = "Actor check"
DESCRIPTION = "Verifies concurrency questions actually use an actor."


def grade(test):
    if "thread-safe" not in test.prompt.lower():
        return None                      # abstain — not my kind of question
    used_actor = "actor " in (test.response or "")
    return Grade(
        score=1.0 if used_actor else 0.0,
        label="actor" if used_actor else "no actor",
    )

Contributing UI

Plugins describe interface as data and the app renders it, so installing one never requires rebuilding the bundle:

from plugin_api import Action, NavItem, Page, Panel, StatRow, Table

def ui_contributions():
    base = "/api/plugins/my_plugin"
    return [
        NavItem(label="My tool", path="/tool", icon="wrench", order=50),
        Page(path="/tool", title="My tool", blocks=[
            StatRow(source=f"{base}/stats"),
            Panel(title="Results", blocks=[
                Table(source=f"{base}/rows"),
                Action(label="Clear", post=f"{base}/clear", style="ghost"),
            ]),
        ]),
    ]

Blocks are StatRow, Table, Markdown, Action and Panel; surfaces are NavItem, Page and SlotPanel. Any block may take inline data or a source URL it fetches live. Paths are namespaced under /x/<slug>/, so plugins cannot collide with each other or shadow a built-in route. Full vocabulary at /docs.

The bundled graders

plugins/answer_key.py compares answers against answers.json files that sit beside the questions, using must_contain, must_contain_any, numbers and regex matchers scored as the fraction satisfied. No key means abstain.

Two rules govern what gets a key, both learned the hard way:

• Verify the value before writing it. tests_py/test_answer_values.py recomputes every numeric key from scratch. It exists because one key was recorded backwards, and shipping it would have failed every correct answer.
• Prefer checks that can only pass wrongly, never fail wrongly. No key uses a negative matcher: this suite is full of questions that require naming the wrong answer — "show why it is not 50%", "quote the exact text" — so a banned substring penalises correct work.

plugins/code_lint.py statically analyses code blocks in answers — syntax, compilation, the signature the prompt demanded, stub bodies, undefined names, unused imports and more. It never executes model output; every check is parse-level, so a broken or hostile snippet cannot affect the grading machine. It abstains on answers with no code.

plugins/response_checks.py ships enabled. It grades any answer — prose, maths, JSON, translation, code — by deriving a rubric from the prompt itself rather than from a hard-coded per-question key. Nothing in it is domain-specific.

It reads the prompt for constraints the answer can be measured against, then scores each as a fraction rather than a pass/fail:

Check	Fires when the prompt…
structure/json	asks for valid JSON — the answer must parse
structure/table	asks for a table
structure/code	names a language or a def, or forbids a code fence
coverage/enumerated	lists numbered deliverables or PART/STAGE blocks
coverage/keyterms	backticks a symbol or quotes a key/heading to emit
constraint/forbidden	bans a symbol ("do not use INIntent")
constraint/length	states one unambiguous word budget
behaviour/abstention	unconditionally requires a refusal or an "I don't know"
quality/degeneration	always — catches looping, truncation, empty output
quality/substance	always — catches one-line non-answers

Only applicable checks count toward the score, so a prompt that never mentions JSON is never scored on JSON. Three deliberate conservatisms keep it from punishing correct work:

• A word budget is enforced only when it unambiguously covers the whole answer. A prompt asking for two summaries of different lengths skips the check rather than failing it.
• A conditional demand ("if the constraints are contradictory, say so") is never scored — whether the condition holds is exactly what the grader cannot determine. Only unconditional demands ("do not guess") count.
• Multi-word quoted spans are ignored as requirements. They are nearly always material being discussed — a line of the source text, or a manipulative phrase the answer is asked to quote back — not something to reproduce.

The report gains a Compliance by check table showing which dimensions the model struggled with across the whole run, and a list of questions with unmet constraints. A low score means the answer did not do what it was told, which is not the same as being wrong — read the answer before treating it as a failure.

How grading behaves

• Abstaining is free. Returning None excludes the question from that grader entirely; it never counts as a zero.
• A question's score is the mean of every grader that scored it. The run's score is the mean of those per-question scores.
• Failed questions are never graded — there is no answer to judge. Use on_test_complete if you want to see failures.
• Grades replace the report's "grade this by hand" placeholder with a table of scores, a letter grade, and each grader's notes. With no graders installed, the report is unchanged from v1.
• Scores appear live in the run feed and in the Reports table.

Failure isolation

A plugin that raises is logged with its slug and skipped for that call only — it stays loaded and its other hooks keep firing. A plugin that fails to import is listed in Settings → Plugins with the error, and everything else still runs. A plugin can never fail a run or stop the server.

Reloading

Settings → Plugins → Reload re-scans the directory, picking up new graders and lifecycle hooks without a restart. Plugins that add HTTP routes need a full restart, since routes are bound when the server starts.

Set ENABLED = False in a plugin to keep the file but stop loading it.

---

CLI

The CLI does not require the frontend to be built.

python auto-test.py -p "Local Engine" -m gemma-4-E2B-it-Q8_0 -t math-code

Flag	Purpose
-h, --help	Usage instructions
-p <provider>	Provider to use (e.g. "Local Engine", "LM Studio")
-m <model>	Model by id or filename
-l, --list	List providers, or models when combined with -p
-s, --suites	List available suites
-t, --test <slug>	Suites to run — repeatable or comma-separated. Omit for every built-in

Reports are written to results/automated_report_<model>.md.

---

Tests

python tests_py/run_all.py     # 227 assertions
cd web && npm test             # 29 assertions

Both run in CI on every push — see .github/workflows/tests.yml.

The Python tests need only requirements-ci.txt, which omits llama-cpp-python: it wants a C compiler, dominates install time, and nothing in the test suite touches local inference. tests_py/test_api.py additionally needs a running server and skips cleanly without one, so the rest stay useful offline. Details in tests_py/README.md.

---

Project structure

.core/                  Diagnostic engine (unchanged)
  providers/            Provider adapters (LM Studio, Local Engine)
  .engine/              Hardware runtimes (MLX, CUDA, ROCm, CPU)
  runner.py             Orchestrates a run and its report
  reporting.py          Markdown report generation
  prompts.py            Facade over suites.py (kept for the runner)
  suites.py             Named suites: discovery, loading, CRUD
  templates/            test-block.md — the per-question report template
server/                 FastAPI backend wrapping the engine
  core_bridge.py        Import shim for the hidden .core package
  api.py                REST endpoints
  run_manager.py        Background runs + server-sent-event streaming
  suite.py              Server layer over the core suite module
  plugins.py            Bridge between server internals and the plugin API
plugins/                Drop-in plugins — _skeleton.py is the starter template
plugin_api.py           Stable types plugins import
plugin_system.py        Plugin discovery and hook dispatch
tests_py/               Python tests — python tests_py/run_all.py
web/                    React + TypeScript interface (Vite)
models/                 Drop .gguf or MLX weights here (auto-discovered)
.core/suites/<slug>/    Built-in suites, read-only
tests/<slug>/           Custom suites, one prompt per .txt
results/                Generated markdown reports
app.py                  Web entrypoint
auto-test.py            CLI entrypoint

---

How a run works

1. Provider selection — providers are registered in .core/providers/, each implementing list_models() and run_prompt(). The Local Engine picks the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA, ROCm on AMD, or a llama-cpp-python CPU fallback.
2. Prompt loading — the selected suites, each in natural filename order, tagged with a qualified <suite>/<file> ID.
3. Execution — one prompt at a time against the selected model. The backend streams each result to the browser as it lands, so nothing is buffered until the end.
4. Reporting — each result is rendered through .core/templates/test-block.md, with a performance summary written at the top once the run finishes.

---

API

The backend is a normal REST API — interactive docs at http://localhost:8765/api/docs.

Endpoint	Purpose
GET /api/providers · GET /api/providers/{name}/models	Discovery
GET /api/suites · GET /api/suites/{slug}	List suites, read one with its questions
POST /api/suites · PUT /api/suites/{slug} · DELETE	Create, rename, remove a custom suite (409 on built-ins)
PUT /api/suites/{slug}/tests	Replace a custom suite's questions (409 on built-ins)
POST /api/suites/{slug}/duplicate	Clone any suite into an editable custom one
POST /api/runs · GET /api/runs/{id}/events · POST /api/runs/{id}/cancel	Start, stream, cancel
GET /api/reports · GET /api/reports/{name}	Saved reports
POST /api/playground	One-off prompt
GET /api/settings · PUT /api/settings	Preferences
GET /api/plugins · POST /api/plugins/reload	Installed plugins
/api/plugins/<slug>/…	Whatever a plugin's register_routes defines

Only one run executes at a time — the local engine loads weights into memory, so overlapping runs would compete for RAM and produce meaningless throughput numbers.

---

Configuration

Variable	Purpose
LM_STUDIO_BASE_URL	LM Studio endpoint (default http://localhost:1234)
AUTO_TEST_TEMPERATURE	Default sampling temperature
AUTO_TEST_PROVIDER	Default provider for CLI runs
LOCAL_LLM_PATHS	os.pathsep-separated extra directories to scan for .gguf weights

Settings changed in the interface persist to .core/user_settings.json and are shared with the CLI. Common LM Studio locations (~/.lmstudio, ~/.lmstudio/models, ~/Library/Application Support/lm-studio/models) are scanned by default.

---

Frontend development

python app.py --no-browser     # terminal 1 — API on :8765
cd web && npm run dev          # terminal 2 — UI on :5173 with hot reload

Vite proxies /api (including the event stream) through to the Python server, so both run from one origin.

---

Building a release

python packaging/build.py --platform all      # the whole release set
python packaging/build.py                     # just the current platform
python packaging/build.py --platform linux    # or windows / macos
python packaging/build.py --skip-web          # reuse an existing web/dist

Output lands in dist/ — .zip for Windows, .tar.gz for macOS and Linux. The script is stdlib-only, so it runs before any dependency is installed, and it can cross-build: a Linux tarball produced on Windows is byte-for-byte equivalent to one produced on Linux, including the executable bit on LM-Gambit.sh (Windows has no execute bit, so the mode is written into the archive metadata rather than read off the filesystem).

Only files git ls-files reports get copied, plus web/dist. Local models, results, settings and virtual environments cannot leak into an archive even if they are sitting in the working tree.

In CI

Workflows exist for both forges. Pushing a v* tag produces a draft release with the three archives attached; workflow_dispatch builds them without creating a release, which is how to test the pipeline before a tag exists.

git tag v2.0.0 && git push origin v2.0.0

	Builds on	Publishes with
.github/workflows/release.yml	windows-latest, ubuntu-latest, macos-latest	softprops/action-gh-release
.gitea/workflows/release.yml	one ubuntu-latest runner	packaging/publish_gitea.py

The Gitea one needs only a single runner because the cross-build makes the archives independent of build host — see packaging/GITEA-SETUP.md for runner registration and the GITEA_TOKEN secret. publish_gitea.py is stdlib-only and idempotent: re-running on the same tag replaces assets rather than duplicating them.

---

Extending

A new provider — add a class in .core/providers/ inheriting from Provider, implement list_models() and run_prompt(), and register it in .core/providers/__init__.py. It appears in the interface automatically.

A new engine runtime — add .core/.engine/.<architecture>/<version>.py defining an EngineRuntime class that inherits from BaseRuntime. The loader picks it up when the hardware matches.

---

Troubleshooting

"Frontend not built" — run cd web && npm install && npm run build. The API and CLI work without it.

No models found — put .gguf files in models/, add a path under Settings, or set LOCAL_LLM_PATHS. For LM Studio, make sure the app is running with its API enabled.

Port already in use — app.py automatically tries the next 20 ports, or pass --port.

Engine errors — check Settings → Engine for the detected runtime, and confirm your hardware drivers and Python dependencies are installed.

---

License

MIT License. See LICENSE for details.