festbench-llm/README.md

# festbench

An LLM benchmark suite that runs entirely in a container. Point it at any
OpenAI-compatible endpoint and it scores writing, code, clinical reasoning,
infrastructure-as-code, long-context retrieval, and hands-on repair of a live
Kubernetes cluster.

Nothing is installed on your machine except the binary. No Python, no OpenTofu,
no Ansible, no kubectl — only a container runtime.

This repository distributes the **binary only**. The source is maintained
privately.

---

## Install

```bash
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.6/festbench_0.3.6_linux_amd64
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.6/SHA256SUMS
sha256sum -c SHA256SUMS
chmod +x festbench_0.3.6_linux_amd64 && sudo mv festbench_0.3.6_linux_amd64 /usr/local/bin/festbench
```

**Requirements:** Linux x86_64, and Docker or Podman. That is all.

Check the machine before your first run — it catches the problems that fail
confusingly rather than clearly:

```bash
festbench --check
```

---

## Flags

| flag | default | purpose |
|---|---|---|
| `--endpoint` | `$BENCH_URL` | OpenAI-compatible API base URL. **Required.** |
| `--suites` | all one-shot suites | comma-separated: `softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench` |
| `--levels` | suite default | thinking levels to sweep: `off,low,medium,xhigh` |
| `--repeat` | 1 | samples per item. **Use 3 or more for any number you intend to quote.** |
| `--label` | — | free text stored with the run, for telling runs apart later |
| `--harness-model` | — | route through a containerised agent, e.g. `github-copilot/claude-sonnet-5:off` |
| `--no-harness` | — | talk straight to `--endpoint` (the default behaviour) |
| `--scenario` | all | scenariobench only: run one scenario |
| `--minutes` | 15 | scenariobench only: per-scenario time budget |
| `--image` | `build.diganthrope.com/pch/festbench:latest` | runner image |
| `--results` | beside the binary | where the JSON and SQLite database land |
| `--runtime` | `auto` | `auto` \| `docker` \| `podman` |
| `-i` | — | guided run: checks prerequisites, then asks what to run |

```bash
# simplest thing that works
festbench --endpoint https://your-llm:8000 --suites softbench

# a real measurement, not a single sample
festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh

# through an agent harness instead of straight to the endpoint
festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off
```

---

## The suites

Six suites. Five send a prompt and score the reply; the sixth puts the model in
front of a real broken cluster.

### softbench — code, prediction, traps

19 items: write code that must pass tests, predict exactly what a program
prints, avoid reasoning traps, and respect forbidden-construct constraints.
Output prediction is the discriminating section — the answers are unambiguous
and a model either understands the semantics or does not.

### prosebench — writing against planted ground truth

10 items scored on adherence and factual control rather than style: word-count
limits, required content, forbidden claims. A model that writes beautifully and
ignores the brief scores badly, which is the point.

### psychbench — clinical reasoning

12 synthetic cases, each built backwards from a known answer with hidden
evidence, a decoy diagnosis, and a required action. Scored on whether the model
finds the evidence and declines to settle on the decoy.

### infrabench — infrastructure as code

10 tasks validated by **real tools**, not regex: OpenTofu, kubectl, kubeconform
and ansible-doc all run against the model's output inside the image. This is why
the image is pinned — the same manifest gets the same verdict everywhere.
Catches things like hallucinated Ansible module names, which pass YAML
validation and fail at run time.

### depthsweep — usable context

Holds a document at a fixed length and moves one planted fact through it, so the
result is a curve rather than a point. The failure that matters is positional:
models often retrieve reliably from the start and end of a long context and lose
the middle. Filler is generated and frozen, so nothing can be answered from
training memory.

### scenariobench — live cluster repair

Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it
with a fictional company's estate, plants one fault, and gives the model a shell
on a jump host with scoped credentials. The model investigates, fixes, and
submits; an automated checker inspects the **cluster**, not the model's report,
and answers only accepted or rejected.

Faults include RBAC denials, unschedulable workloads, expired certificates, DNS
policy mismatches, an OOM kill whose cause is a different workload, a broken
kubelet runtime endpoint, and a systemd unit failing at boot.

---

## Options that change what you measure

### Thinking levels

`--levels off,low,medium,xhigh` sweeps the model's reasoning effort. The cost is
not linear and is worth knowing before you start — measured on Qwen3.8-27B:

| suite | `off` | `low` | `medium` | `xhigh` |
|---|---|---|---|---|
| softbench | 1.2s | 4.5s | 4.4s | 6.1s |
| infrabench | 2.6s | 12.4s | 18.7s | **97.1s** |
| prosebench | 7.1s | 17.0s | 23.7s | 39.6s |
| psychbench | 11.7s | 32.5s | 55.1s | **117.0s** |

*(seconds per item)*

`xhigh` can cost **37×** `off` on the same tasks. Whether it buys anything is
the question the sweep answers — on some suites it gains a point, on others
nothing at all.

### Harness on or off

**Off (default)** — the suite talks straight to `--endpoint`. This measures the
model.

**On** — `--harness-model provider/model` routes every request through a
containerised pi.dev agent with its own tools. This measures the model *in an
agent loop*, which is closer to how it would actually be used, and lets a run
reach any provider the harness is logged in to (GitHub Copilot, Anthropic,
OpenAI) rather than only your endpoint.

Thinking levels work differently in each mode. Against a local endpoint the
level travels in the chat template and is sweepable. Through the harness it
rides on the model name — `claude-sonnet-5:high` — and a run without a suffix is
recorded as `default` rather than claiming a level nobody set.

---

## How long it takes

Measured, not estimated. Wall-clock for a complete run of each suite at
`--repeat 1`:

| suite | items | `off` | `xhigh` |
|---|---|---|---|
| softbench | 19 | **2 min** | 4 min |
| infrabench | 10 | **3 min** | 24 min |
| prosebench | 10 | **6 min** | 7 min |
| psychbench | 12 | **12 min** | 23 min |
| depthsweep | 10 | **13 min** | 13 min |
| all five | 61 | **~35 min** | ~70 min |

**scenariobench is different.** It is time-boxed rather than item-counted: a
scenario runs until the model submits an accepted fix or the budget expires.
Across 39 runs the mean was **9 minutes** per scenario (range 1–23), plus 2–3
minutes to build and tear down the cluster. All seven scenarios at the default
15-minute budget is a **1–2 hour** job.

Timings scale with `--repeat`, and depthsweep scales with context size — the
figures above are at its default, and a 100K-token run costs roughly 3 minutes
per item because the whole context must be processed before a single token comes
back.

---

## Results

Everything lands in `results/` beside the binary: one JSON file per suite and a
SQLite database accumulating every run, with the model, the endpoint, the
thinking level, per-item timings, token counts, and the digest of the image that
produced the score.

Two things the tool will refuse to do, both because a wrong number is worse than
no number: it will not run against an endpoint that is not answering, and it
will not run infrabench with validators that cannot actually validate.