LM-Gambit/README.md

LM-Gambit v1.0.0: Automated LLM Diagnostic Suite

Version 1.0.0

LM-Gambit is a modular, extensible framework for benchmarking and analyzing local or remote Large Language Models (LLMs). It supports both a command-line interface (CLI) and a Tkinter-based GUI, enabling rapid, repeatable evaluation of LLMs using customizable prompt suites.

---

Features

• Automated Testing: Run a suite of prompt-based diagnostics against any supported LLM provider.
• Provider System: Out-of-the-box support for LM Studio (API) and a native Local Engine (hardware-optimized, supports Apple Silicon, CUDA, ROCm, CPU).
• Extensible: Add new providers or engine runtimes with minimal code changes.
• GUI & CLI: Choose between a full-featured GUI or a fast CLI for scripting and automation.
• Customizable Reports: Generates detailed Markdown reports with performance metrics and qualitative analysis sections.
• Prompt Management: Prompts are simple .txt files, editable in the GUI or any text editor.

---

Installation

Requirements:

• Python 3.10+
• macOS, Linux, or Windows (Apple Silicon, NVIDIA, AMD, or CPU-only supported)

Install dependencies:

python -m pip install -r requirements.txt

(Optional) For Apple Silicon (MLX), ensure mlx-lm is installed. For CUDA/ROCm, ensure your system drivers are set up.

---

Quick Start

CLI Usage

Run all tests with the default provider/model:

python auto-test.py

Options:

• -h, --help — Show usage instructions
• -p <provider> — Select provider (e.g., "Local Engine", "LM Studio")
• -m <model> — Select model by ID or filename
• -l, --list — List available providers or models
• -t <test> — (Reserved) Specify a test suite

Example:

python auto-test.py -p "Local Engine" -m Qwen3-Coder-30B.gguf

Reports are saved to results/automated_report_<model>.md.

GUI Usage

python index.py

GUI Features:

• Provider/model dropdowns with auto-discovery
• Adjustable sampling temperature
• Run, refresh, open prompts/results, and view latest report
• Live log of test progress
• Settings dialog for managing model search paths

---

Project Structure

• .core/ — Core logic, providers, engine loader, reporting, templates
  • providers/ — Provider adapters (LM Studio, Local Engine, etc.)
  • .engine/ — Hardware-specific runtime implementations (MLX, CUDA, ROCm, CPU)
  • runner.py — Orchestrates test runs, progress, and reporting
  • reporting.py — Markdown report generation, code-fence hygiene
  • prompts.py — Loads and parses prompt files from tests/
  • templates/test-block.md — Customizable template for each test result
• models/ — Drop-in directory for .gguf or MLX-compatible weights (auto-discovered)
• tests/ — One prompt per .txt file (edit or add your own)
• results/ — Markdown reports generated after each run
• auto-test.py — CLI entrypoint
• index.py — Tkinter GUI entrypoint

---

How the Core Engine Works

1. Provider Selection:

   • Providers (e.g., LM Studio, Local Engine) are registered in .core/providers/.
   • Each provider implements list_models() and run_prompt().
   • The Local Engine auto-selects the best runtime for your hardware (MLX, CUDA, ROCm, or CPU fallback).

2. Prompt Loading:

   • Prompts are loaded from the tests/ directory (one .txt file per test).
   • Each prompt file's first non-empty line is used as the test title.

3. Test Execution:

   • For each prompt, the selected provider/model is used to generate a response.
   • Results are streamed to the CLI or GUI with live progress updates.

4. Reporting:

   • Each test result is rendered using a Markdown template (.core/templates/test-block.md).
   • Performance metrics (tokens/sec, time to first token, etc.) are included.
   • A summary and qualitative analysis section are generated at the top of the report.

---

Configuration & Environment Variables

You can customize behavior via environment variables:

Variable	Purpose
LM_STUDIO_BASE_URL	Override LM Studio API endpoint (default: http://localhost:1234)
AUTO_TEST_TEMPERATURE	Default sampling temperature for test runs
AUTO_TEST_PROVIDER	Default provider for CLI runs
LOCAL_LLM_PATHS	:-separated list of extra directories for .gguf models

Model search paths can also be managed via the GUI settings dialog (persisted in .core/user_settings.json).

---

Extending LM-Gambit

Add a new provider:

1. Create a new class in .core/providers/ inheriting from Provider (see base.py).
2. Implement list_models() and run_prompt().
3. Register your provider in .core/providers/__init__.py.

Add a new engine runtime:

1. Add a new Python file under .core/.engine/.<architecture>/<version>.py.
2. Implement an EngineRuntime class inheriting from BaseRuntime.
3. The engine loader will auto-detect and use your runtime if the hardware matches.

---

Customizing Prompts & Reports in LM-Gambit

• Add/edit prompt .txt files in tests/ (first non-empty line is the title).
• Edit .core/templates/test-block.md to change the report format.

---

Troubleshooting

• No models found?
  • Ensure your .gguf files are in models/ or listed in LOCAL_LLM_PATHS.
  • For LM Studio, ensure the app is running and the API is enabled.
• Engine errors?
  • Check your hardware drivers and Python dependencies.
• GUI not launching?
  • Ensure tkinter is installed and available in your Python environment.

---

License

MIT License. See LICENSE for details.

• LM_STUDIO_BASE_URL – override the default http://localhost:1234 endpoint for the LM Studio provider.
• AUTO_TEST_TEMPERATURE – default sampling temperature for test runs.
• AUTO_TEST_PROVIDER – default provider name for CLI runs.
• LOCAL_LLM_PATHS – optional os.pathsep-separated list of extra directories to scan for .gguf weights.

Templates can be customized by editing .core/templates/test-block.md.

Local Engine provider

• Place .gguf (llama.cpp) weights inside models/ or add extra directories via the GUI settings dialog / LOCAL_LLM_PATHS environment variable.
• Paths include common defaults (for example ~/.lmstudio, ~/.lmstudio/models, or ~/Library/Application Support/lm-studio/models on macOS) so LM Studio downloads are discovered automatically.
• The engine loader automatically selects the best runtime for your hardware: MLX on Apple Silicon, CUDA on NVIDIA GPUs, ROCm on AMD GPUs, or a CPU fallback via llama-cpp-python.
• Settings are saved in .core/user_settings.json, keeping CLI and GUI runs in sync.

Extending Providers in LM-Gambit

Provider adapters live under .core/providers/ and are registered in .core/providers/__init__.py. Each provider implements list_models and run_prompt to integrate with the runner and GUI, while hardware-specific runtimes reside under .core/.engine/.