festbench-llm/README.md

festbench

An LLM benchmark suite that runs entirely in a container. Point it at any OpenAI-compatible endpoint and it scores writing, code, clinical reasoning, infrastructure-as-code, long-context retrieval, and hands-on repair of a live Kubernetes cluster.

Nothing is installed on your machine except the binary. No Python, no OpenTofu, no Ansible, no kubectl — only a container runtime.

This repository distributes the binary only. The source is maintained privately.

---

Install

curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.2/festbench_0.3.2_linux_amd64
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.2/SHA256SUMS
sha256sum -c SHA256SUMS
chmod +x festbench_0.3.2_linux_amd64 && sudo mv festbench_0.3.2_linux_amd64 /usr/local/bin/festbench

Requirements: Linux x86_64, and Docker or Podman. That is all.

Check the machine before your first run — it catches the problems that fail confusingly rather than clearly:

festbench --check

---

Flags

flag	default	purpose
--endpoint	$BENCH_URL	OpenAI-compatible API base URL. Required.
--suites	all one-shot suites	comma-separated: softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench
--levels	suite default	thinking levels to sweep: off,low,medium,xhigh
--repeat	1	samples per item. Use 3 or more for any number you intend to quote.
--label	—	free text stored with the run, for telling runs apart later
--harness-model	—	route through a containerised agent, e.g. github-copilot/claude-sonnet-5:off
--no-harness	—	talk straight to --endpoint (the default behaviour)
--scenario	all	scenariobench only: run one scenario
--minutes	15	scenariobench only: per-scenario time budget
--image	build.diganthrope.com/pch/festbench:latest	runner image
--results	beside the binary	where the JSON and SQLite database land
--runtime	auto	auto | docker | podman
-i	—	guided run: checks prerequisites, then asks what to run

# simplest thing that works
festbench --endpoint https://your-llm:8000 --suites softbench

# a real measurement, not a single sample
festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh

# through an agent harness instead of straight to the endpoint
festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off

---

The suites

Six suites. Five send a prompt and score the reply; the sixth puts the model in front of a real broken cluster.

softbench — code, prediction, traps

19 items: write code that must pass tests, predict exactly what a program prints, avoid reasoning traps, and respect forbidden-construct constraints. Output prediction is the discriminating section — the answers are unambiguous and a model either understands the semantics or does not.

prosebench — writing against planted ground truth

10 items scored on adherence and factual control rather than style: word-count limits, required content, forbidden claims. A model that writes beautifully and ignores the brief scores badly, which is the point.

psychbench — clinical reasoning

12 synthetic cases, each built backwards from a known answer with hidden evidence, a decoy diagnosis, and a required action. Scored on whether the model finds the evidence and declines to settle on the decoy.

infrabench — infrastructure as code

10 tasks validated by real tools, not regex: OpenTofu, kubectl, kubeconform and ansible-doc all run against the model's output inside the image. This is why the image is pinned — the same manifest gets the same verdict everywhere. Catches things like hallucinated Ansible module names, which pass YAML validation and fail at run time.

depthsweep — usable context

Holds a document at a fixed length and moves one planted fact through it, so the result is a curve rather than a point. The failure that matters is positional: models often retrieve reliably from the start and end of a long context and lose the middle. Filler is generated and frozen, so nothing can be answered from training memory.

scenariobench — live cluster repair

Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it with a fictional company's estate, plants one fault, and gives the model a shell on a jump host with scoped credentials. The model investigates, fixes, and submits; an automated checker inspects the cluster, not the model's report, and answers only accepted or rejected.

Faults include RBAC denials, unschedulable workloads, expired certificates, DNS policy mismatches, an OOM kill whose cause is a different workload, a broken kubelet runtime endpoint, and a systemd unit failing at boot.

---

Options that change what you measure

Thinking levels

--levels off,low,medium,xhigh sweeps the model's reasoning effort. The cost is not linear and is worth knowing before you start — measured on Qwen3.8-27B:

suite	off	low	medium	xhigh
softbench	1.2s	4.5s	4.4s	6.1s
infrabench	2.6s	12.4s	18.7s	97.1s
prosebench	7.1s	17.0s	23.7s	39.6s
psychbench	11.7s	32.5s	55.1s	117.0s

(seconds per item)

xhigh can cost 37× off on the same tasks. Whether it buys anything is the question the sweep answers — on some suites it gains a point, on others nothing at all.

Harness on or off

Off (default) — the suite talks straight to --endpoint. This measures the model.

On — --harness-model provider/model routes every request through a containerised pi.dev agent with its own tools. This measures the model in an agent loop, which is closer to how it would actually be used, and lets a run reach any provider the harness is logged in to (GitHub Copilot, Anthropic, OpenAI) rather than only your endpoint.

Thinking levels work differently in each mode. Against a local endpoint the level travels in the chat template and is sweepable. Through the harness it rides on the model name — claude-sonnet-5:high — and a run without a suffix is recorded as default rather than claiming a level nobody set.

---

How long it takes

Measured, not estimated. Wall-clock for a complete run of each suite at --repeat 1:

suite	items	off	xhigh
softbench	19	2 min	4 min
infrabench	10	3 min	24 min
prosebench	10	6 min	7 min
psychbench	12	12 min	23 min
depthsweep	10	13 min	13 min
all five	61	~35 min	~70 min

scenariobench is different. It is time-boxed rather than item-counted: a scenario runs until the model submits an accepted fix or the budget expires. Across 39 runs the mean was 9 minutes per scenario (range 1–23), plus 2–3 minutes to build and tear down the cluster. All seven scenarios at the default 15-minute budget is a 1–2 hour job.

Timings scale with --repeat, and depthsweep scales with context size — the figures above are at its default, and a 100K-token run costs roughly 3 minutes per item because the whole context must be processed before a single token comes back.

---

Results

Everything lands in results/ beside the binary: one JSON file per suite and a SQLite database accumulating every run, with the model, the endpoint, the thinking level, per-item timings, token counts, and the digest of the image that produced the score.

Two things the tool will refuse to do, both because a wrong number is worse than no number: it will not run against an endpoint that is not answering, and it will not run infrabench with validators that cannot actually validate.