7.7 KiB
festbench
An LLM benchmark suite that runs entirely in a container. Point it at any OpenAI-compatible endpoint and it scores writing, code, clinical reasoning, infrastructure-as-code, long-context retrieval, and hands-on repair of a live Kubernetes cluster.
Nothing is installed on your machine except the binary. No Python, no OpenTofu, no Ansible, no kubectl — only a container runtime.
This repository distributes the binary only. The source is maintained privately.
Install
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.1/festbench_0.3.1_linux_amd64
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.1/SHA256SUMS
sha256sum -c SHA256SUMS
chmod +x festbench_0.3.1_linux_amd64 && sudo mv festbench_0.3.1_linux_amd64 /usr/local/bin/festbench
Requirements: Linux x86_64, and Docker or Podman. That is all.
Check the machine before your first run — it catches the problems that fail confusingly rather than clearly:
festbench --check
Flags
| flag | default | purpose |
|---|---|---|
--endpoint |
$BENCH_URL |
OpenAI-compatible API base URL. Required. |
--suites |
all one-shot suites | comma-separated: softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench |
--levels |
suite default | thinking levels to sweep: off,low,medium,xhigh |
--repeat |
1 | samples per item. Use 3 or more for any number you intend to quote. |
--label |
— | free text stored with the run, for telling runs apart later |
--harness-model |
— | route through a containerised agent, e.g. github-copilot/claude-sonnet-5:off |
--no-harness |
— | talk straight to --endpoint (the default behaviour) |
--scenario |
all | scenariobench only: run one scenario |
--minutes |
15 | scenariobench only: per-scenario time budget |
--image |
build.diganthrope.com/pch/festbench:latest |
runner image |
--results |
beside the binary | where the JSON and SQLite database land |
--runtime |
auto |
auto | docker | podman |
-i |
— | guided run: checks prerequisites, then asks what to run |
# simplest thing that works
festbench --endpoint https://your-llm:8000 --suites softbench
# a real measurement, not a single sample
festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh
# through an agent harness instead of straight to the endpoint
festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off
The suites
Six suites. Five send a prompt and score the reply; the sixth puts the model in front of a real broken cluster.
softbench — code, prediction, traps
19 items: write code that must pass tests, predict exactly what a program prints, avoid reasoning traps, and respect forbidden-construct constraints. Output prediction is the discriminating section — the answers are unambiguous and a model either understands the semantics or does not.
prosebench — writing against planted ground truth
10 items scored on adherence and factual control rather than style: word-count limits, required content, forbidden claims. A model that writes beautifully and ignores the brief scores badly, which is the point.
psychbench — clinical reasoning
12 synthetic cases, each built backwards from a known answer with hidden evidence, a decoy diagnosis, and a required action. Scored on whether the model finds the evidence and declines to settle on the decoy.
infrabench — infrastructure as code
10 tasks validated by real tools, not regex: OpenTofu, kubectl, kubeconform and ansible-doc all run against the model's output inside the image. This is why the image is pinned — the same manifest gets the same verdict everywhere. Catches things like hallucinated Ansible module names, which pass YAML validation and fail at run time.
depthsweep — usable context
Holds a document at a fixed length and moves one planted fact through it, so the result is a curve rather than a point. The failure that matters is positional: models often retrieve reliably from the start and end of a long context and lose the middle. Filler is generated and frozen, so nothing can be answered from training memory.
scenariobench — live cluster repair
Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it with a fictional company's estate, plants one fault, and gives the model a shell on a jump host with scoped credentials. The model investigates, fixes, and submits; an automated checker inspects the cluster, not the model's report, and answers only accepted or rejected.
Faults include RBAC denials, unschedulable workloads, expired certificates, DNS policy mismatches, an OOM kill whose cause is a different workload, a broken kubelet runtime endpoint, and a systemd unit failing at boot.
Options that change what you measure
Thinking levels
--levels off,low,medium,xhigh sweeps the model's reasoning effort. The cost is
not linear and is worth knowing before you start — measured on Qwen3.8-27B:
| suite | off |
low |
medium |
xhigh |
|---|---|---|---|---|
| softbench | 1.2s | 4.5s | 4.4s | 6.1s |
| infrabench | 2.6s | 12.4s | 18.7s | 97.1s |
| prosebench | 7.1s | 17.0s | 23.7s | 39.6s |
| psychbench | 11.7s | 32.5s | 55.1s | 117.0s |
(seconds per item)
xhigh can cost 37× off on the same tasks. Whether it buys anything is
the question the sweep answers — on some suites it gains a point, on others
nothing at all.
Harness on or off
Off (default) — the suite talks straight to --endpoint. This measures the
model.
On — --harness-model provider/model routes every request through a
containerised pi.dev agent with its own tools. This measures the model in an
agent loop, which is closer to how it would actually be used, and lets a run
reach any provider the harness is logged in to (GitHub Copilot, Anthropic,
OpenAI) rather than only your endpoint.
Thinking levels work differently in each mode. Against a local endpoint the
level travels in the chat template and is sweepable. Through the harness it
rides on the model name — claude-sonnet-5:high — and a run without a suffix is
recorded as default rather than claiming a level nobody set.
How long it takes
Measured, not estimated. Wall-clock for a complete run of each suite at
--repeat 1:
| suite | items | off |
xhigh |
|---|---|---|---|
| softbench | 19 | 2 min | 4 min |
| infrabench | 10 | 3 min | 24 min |
| prosebench | 10 | 6 min | 7 min |
| psychbench | 12 | 12 min | 23 min |
| depthsweep | 10 | 13 min | 13 min |
| all five | 61 | ~35 min | ~70 min |
scenariobench is different. It is time-boxed rather than item-counted: a scenario runs until the model submits an accepted fix or the budget expires. Across 39 runs the mean was 9 minutes per scenario (range 1–23), plus 2–3 minutes to build and tear down the cluster. All seven scenarios at the default 15-minute budget is a 1–2 hour job.
Timings scale with --repeat, and depthsweep scales with context size — the
figures above are at its default, and a 100K-token run costs roughly 3 minutes
per item because the whole context must be processed before a single token comes
back.
Results
Everything lands in results/ beside the binary: one JSON file per suite and a
SQLite database accumulating every run, with the model, the endpoint, the
thinking level, per-item timings, token counts, and the digest of the image that
produced the score.
Two things the tool will refuse to do, both because a wrong number is worse than no number: it will not run against an endpoint that is not answering, and it will not run infrabench with validators that cannot actually validate.