From 28b8e682a8e6191bcb9d70e237922dc4999a3036 Mon Sep 17 00:00:00 2001 From: PeterChrz Date: Fri, 11 Sep 2026 22:44:19 -0400 Subject: [PATCH] festbench: install instructions and suite documentation Binary releases only. The source is maintained in a private repository. --- README.md | 195 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..2076e12 --- /dev/null +++ b/README.md @@ -0,0 +1,195 @@ +# festbench + +An LLM benchmark suite that runs entirely in a container. Point it at any +OpenAI-compatible endpoint and it scores writing, code, clinical reasoning, +infrastructure-as-code, long-context retrieval, and hands-on repair of a live +Kubernetes cluster. + +Nothing is installed on your machine except the binary. No Python, no OpenTofu, +no Ansible, no kubectl — only a container runtime. + +This repository distributes the **binary only**. The source is maintained +privately. + +--- + +## Install + +```bash +curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.0/festbench_0.3.0_linux_amd64 +curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.0/SHA256SUMS +sha256sum -c SHA256SUMS +chmod +x festbench_0.3.0_linux_amd64 && sudo mv festbench_0.3.0_linux_amd64 /usr/local/bin/festbench +``` + +**Requirements:** Linux x86_64, and Docker or Podman. That is all. + +Check the machine before your first run — it catches the problems that fail +confusingly rather than clearly: + +```bash +festbench --check +``` + +--- + +## Flags + +| flag | default | purpose | +|---|---|---| +| `--endpoint` | `$BENCH_URL` | OpenAI-compatible API base URL. **Required.** | +| `--suites` | all one-shot suites | comma-separated: `softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench` | +| `--levels` | suite default | thinking levels to sweep: `off,low,medium,xhigh` | +| `--repeat` | 1 | samples per item. **Use 3 or more for any number you intend to quote.** | +| `--label` | — | free text stored with the run, for telling runs apart later | +| `--harness-model` | — | route through a containerised agent, e.g. `github-copilot/claude-sonnet-5:off` | +| `--no-harness` | — | talk straight to `--endpoint` (the default behaviour) | +| `--scenario` | all | scenariobench only: run one scenario | +| `--minutes` | 15 | scenariobench only: per-scenario time budget | +| `--image` | `build.diganthrope.com/pch/festbench:latest` | runner image | +| `--results` | beside the binary | where the JSON and SQLite database land | +| `--runtime` | `auto` | `auto` \| `docker` \| `podman` | +| `-i` | — | guided run: checks prerequisites, then asks what to run | + +```bash +# simplest thing that works +festbench --endpoint https://your-llm:8000 --suites softbench + +# a real measurement, not a single sample +festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh + +# through an agent harness instead of straight to the endpoint +festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off +``` + +--- + +## The suites + +Six suites. Five send a prompt and score the reply; the sixth puts the model in +front of a real broken cluster. + +### softbench — code, prediction, traps + +19 items: write code that must pass tests, predict exactly what a program +prints, avoid reasoning traps, and respect forbidden-construct constraints. +Output prediction is the discriminating section — the answers are unambiguous +and a model either understands the semantics or does not. + +### prosebench — writing against planted ground truth + +10 items scored on adherence and factual control rather than style: word-count +limits, required content, forbidden claims. A model that writes beautifully and +ignores the brief scores badly, which is the point. + +### psychbench — clinical reasoning + +12 synthetic cases, each built backwards from a known answer with hidden +evidence, a decoy diagnosis, and a required action. Scored on whether the model +finds the evidence and declines to settle on the decoy. + +### infrabench — infrastructure as code + +10 tasks validated by **real tools**, not regex: OpenTofu, kubectl, kubeconform +and ansible-doc all run against the model's output inside the image. This is why +the image is pinned — the same manifest gets the same verdict everywhere. +Catches things like hallucinated Ansible module names, which pass YAML +validation and fail at run time. + +### depthsweep — usable context + +Holds a document at a fixed length and moves one planted fact through it, so the +result is a curve rather than a point. The failure that matters is positional: +models often retrieve reliably from the start and end of a long context and lose +the middle. Filler is generated and frozen, so nothing can be answered from +training memory. + +### scenariobench — live cluster repair + +Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it +with a fictional company's estate, plants one fault, and gives the model a shell +on a jump host with scoped credentials. The model investigates, fixes, and +submits; an automated checker inspects the **cluster**, not the model's report, +and answers only accepted or rejected. + +Faults include RBAC denials, unschedulable workloads, expired certificates, DNS +policy mismatches, an OOM kill whose cause is a different workload, a broken +kubelet runtime endpoint, and a systemd unit failing at boot. + +--- + +## Options that change what you measure + +### Thinking levels + +`--levels off,low,medium,xhigh` sweeps the model's reasoning effort. The cost is +not linear and is worth knowing before you start — measured on Qwen3.8-27B: + +| suite | `off` | `low` | `medium` | `xhigh` | +|---|---|---|---|---| +| softbench | 1.2s | 4.5s | 4.4s | 6.1s | +| infrabench | 2.6s | 12.4s | 18.7s | **97.1s** | +| prosebench | 7.1s | 17.0s | 23.7s | 39.6s | +| psychbench | 11.7s | 32.5s | 55.1s | **117.0s** | + +*(seconds per item)* + +`xhigh` can cost **37×** `off` on the same tasks. Whether it buys anything is +the question the sweep answers — on some suites it gains a point, on others +nothing at all. + +### Harness on or off + +**Off (default)** — the suite talks straight to `--endpoint`. This measures the +model. + +**On** — `--harness-model provider/model` routes every request through a +containerised pi.dev agent with its own tools. This measures the model *in an +agent loop*, which is closer to how it would actually be used, and lets a run +reach any provider the harness is logged in to (GitHub Copilot, Anthropic, +OpenAI) rather than only your endpoint. + +Thinking levels work differently in each mode. Against a local endpoint the +level travels in the chat template and is sweepable. Through the harness it +rides on the model name — `claude-sonnet-5:high` — and a run without a suffix is +recorded as `default` rather than claiming a level nobody set. + +--- + +## How long it takes + +Measured, not estimated. Wall-clock for a complete run of each suite at +`--repeat 1`: + +| suite | items | `off` | `xhigh` | +|---|---|---|---| +| softbench | 19 | **2 min** | 4 min | +| infrabench | 10 | **3 min** | 24 min | +| prosebench | 10 | **6 min** | 7 min | +| psychbench | 12 | **12 min** | 23 min | +| depthsweep | 10 | **13 min** | 13 min | +| all five | 61 | **~35 min** | ~70 min | + +**scenariobench is different.** It is time-boxed rather than item-counted: a +scenario runs until the model submits an accepted fix or the budget expires. +Across 39 runs the mean was **9 minutes** per scenario (range 1–23), plus 2–3 +minutes to build and tear down the cluster. All seven scenarios at the default +15-minute budget is a **1–2 hour** job. + +Timings scale with `--repeat`, and depthsweep scales with context size — the +figures above are at its default, and a 100K-token run costs roughly 3 minutes +per item because the whole context must be processed before a single token comes +back. + +--- + +## Results + +Everything lands in `results/` beside the binary: one JSON file per suite and a +SQLite database accumulating every run, with the model, the endpoint, the +thinking level, per-item timings, token counts, and the digest of the image that +produced the score. + +Two things the tool will refuse to do, both because a wrong number is worse than +no number: it will not run against an endpoint that is not answering, and it +will not run infrabench with validators that cannot actually validate.