195 lines
7.7 KiB
Markdown
195 lines
7.7 KiB
Markdown
# festbench
|
||
|
||
An LLM benchmark suite that runs entirely in a container. Point it at any
|
||
OpenAI-compatible endpoint and it scores writing, code, clinical reasoning,
|
||
infrastructure-as-code, long-context retrieval, and hands-on repair of a live
|
||
Kubernetes cluster.
|
||
|
||
Nothing is installed on your machine except the binary. No Python, no OpenTofu,
|
||
no Ansible, no kubectl — only a container runtime.
|
||
|
||
This repository distributes the **binary only**. The source is maintained
|
||
privately.
|
||
|
||
---
|
||
|
||
## Install
|
||
|
||
```bash
|
||
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.6/festbench_0.3.6_linux_amd64
|
||
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.6/SHA256SUMS
|
||
sha256sum -c SHA256SUMS
|
||
chmod +x festbench_0.3.6_linux_amd64 && sudo mv festbench_0.3.6_linux_amd64 /usr/local/bin/festbench
|
||
```
|
||
|
||
**Requirements:** Linux x86_64, and Docker or Podman. That is all.
|
||
|
||
Check the machine before your first run — it catches the problems that fail
|
||
confusingly rather than clearly:
|
||
|
||
```bash
|
||
festbench --check
|
||
```
|
||
|
||
---
|
||
|
||
## Flags
|
||
|
||
| flag | default | purpose |
|
||
|---|---|---|
|
||
| `--endpoint` | `$BENCH_URL` | OpenAI-compatible API base URL. **Required.** |
|
||
| `--suites` | all one-shot suites | comma-separated: `softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench` |
|
||
| `--levels` | suite default | thinking levels to sweep: `off,low,medium,xhigh` |
|
||
| `--repeat` | 1 | samples per item. **Use 3 or more for any number you intend to quote.** |
|
||
| `--label` | — | free text stored with the run, for telling runs apart later |
|
||
| `--harness-model` | — | route through a containerised agent, e.g. `github-copilot/claude-sonnet-5:off` |
|
||
| `--no-harness` | — | talk straight to `--endpoint` (the default behaviour) |
|
||
| `--scenario` | all | scenariobench only: run one scenario |
|
||
| `--minutes` | 15 | scenariobench only: per-scenario time budget |
|
||
| `--image` | `build.diganthrope.com/pch/festbench:latest` | runner image |
|
||
| `--results` | beside the binary | where the JSON and SQLite database land |
|
||
| `--runtime` | `auto` | `auto` \| `docker` \| `podman` |
|
||
| `-i` | — | guided run: checks prerequisites, then asks what to run |
|
||
|
||
```bash
|
||
# simplest thing that works
|
||
festbench --endpoint https://your-llm:8000 --suites softbench
|
||
|
||
# a real measurement, not a single sample
|
||
festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh
|
||
|
||
# through an agent harness instead of straight to the endpoint
|
||
festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off
|
||
```
|
||
|
||
---
|
||
|
||
## The suites
|
||
|
||
Six suites. Five send a prompt and score the reply; the sixth puts the model in
|
||
front of a real broken cluster.
|
||
|
||
### softbench — code, prediction, traps
|
||
|
||
19 items: write code that must pass tests, predict exactly what a program
|
||
prints, avoid reasoning traps, and respect forbidden-construct constraints.
|
||
Output prediction is the discriminating section — the answers are unambiguous
|
||
and a model either understands the semantics or does not.
|
||
|
||
### prosebench — writing against planted ground truth
|
||
|
||
10 items scored on adherence and factual control rather than style: word-count
|
||
limits, required content, forbidden claims. A model that writes beautifully and
|
||
ignores the brief scores badly, which is the point.
|
||
|
||
### psychbench — clinical reasoning
|
||
|
||
12 synthetic cases, each built backwards from a known answer with hidden
|
||
evidence, a decoy diagnosis, and a required action. Scored on whether the model
|
||
finds the evidence and declines to settle on the decoy.
|
||
|
||
### infrabench — infrastructure as code
|
||
|
||
10 tasks validated by **real tools**, not regex: OpenTofu, kubectl, kubeconform
|
||
and ansible-doc all run against the model's output inside the image. This is why
|
||
the image is pinned — the same manifest gets the same verdict everywhere.
|
||
Catches things like hallucinated Ansible module names, which pass YAML
|
||
validation and fail at run time.
|
||
|
||
### depthsweep — usable context
|
||
|
||
Holds a document at a fixed length and moves one planted fact through it, so the
|
||
result is a curve rather than a point. The failure that matters is positional:
|
||
models often retrieve reliably from the start and end of a long context and lose
|
||
the middle. Filler is generated and frozen, so nothing can be answered from
|
||
training memory.
|
||
|
||
### scenariobench — live cluster repair
|
||
|
||
Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it
|
||
with a fictional company's estate, plants one fault, and gives the model a shell
|
||
on a jump host with scoped credentials. The model investigates, fixes, and
|
||
submits; an automated checker inspects the **cluster**, not the model's report,
|
||
and answers only accepted or rejected.
|
||
|
||
Faults include RBAC denials, unschedulable workloads, expired certificates, DNS
|
||
policy mismatches, an OOM kill whose cause is a different workload, a broken
|
||
kubelet runtime endpoint, and a systemd unit failing at boot.
|
||
|
||
---
|
||
|
||
## Options that change what you measure
|
||
|
||
### Thinking levels
|
||
|
||
`--levels off,low,medium,xhigh` sweeps the model's reasoning effort. The cost is
|
||
not linear and is worth knowing before you start — measured on Qwen3.8-27B:
|
||
|
||
| suite | `off` | `low` | `medium` | `xhigh` |
|
||
|---|---|---|---|---|
|
||
| softbench | 1.2s | 4.5s | 4.4s | 6.1s |
|
||
| infrabench | 2.6s | 12.4s | 18.7s | **97.1s** |
|
||
| prosebench | 7.1s | 17.0s | 23.7s | 39.6s |
|
||
| psychbench | 11.7s | 32.5s | 55.1s | **117.0s** |
|
||
|
||
*(seconds per item)*
|
||
|
||
`xhigh` can cost **37×** `off` on the same tasks. Whether it buys anything is
|
||
the question the sweep answers — on some suites it gains a point, on others
|
||
nothing at all.
|
||
|
||
### Harness on or off
|
||
|
||
**Off (default)** — the suite talks straight to `--endpoint`. This measures the
|
||
model.
|
||
|
||
**On** — `--harness-model provider/model` routes every request through a
|
||
containerised pi.dev agent with its own tools. This measures the model *in an
|
||
agent loop*, which is closer to how it would actually be used, and lets a run
|
||
reach any provider the harness is logged in to (GitHub Copilot, Anthropic,
|
||
OpenAI) rather than only your endpoint.
|
||
|
||
Thinking levels work differently in each mode. Against a local endpoint the
|
||
level travels in the chat template and is sweepable. Through the harness it
|
||
rides on the model name — `claude-sonnet-5:high` — and a run without a suffix is
|
||
recorded as `default` rather than claiming a level nobody set.
|
||
|
||
---
|
||
|
||
## How long it takes
|
||
|
||
Measured, not estimated. Wall-clock for a complete run of each suite at
|
||
`--repeat 1`:
|
||
|
||
| suite | items | `off` | `xhigh` |
|
||
|---|---|---|---|
|
||
| softbench | 19 | **2 min** | 4 min |
|
||
| infrabench | 10 | **3 min** | 24 min |
|
||
| prosebench | 10 | **6 min** | 7 min |
|
||
| psychbench | 12 | **12 min** | 23 min |
|
||
| depthsweep | 10 | **13 min** | 13 min |
|
||
| all five | 61 | **~35 min** | ~70 min |
|
||
|
||
**scenariobench is different.** It is time-boxed rather than item-counted: a
|
||
scenario runs until the model submits an accepted fix or the budget expires.
|
||
Across 39 runs the mean was **9 minutes** per scenario (range 1–23), plus 2–3
|
||
minutes to build and tear down the cluster. All seven scenarios at the default
|
||
15-minute budget is a **1–2 hour** job.
|
||
|
||
Timings scale with `--repeat`, and depthsweep scales with context size — the
|
||
figures above are at its default, and a 100K-token run costs roughly 3 minutes
|
||
per item because the whole context must be processed before a single token comes
|
||
back.
|
||
|
||
---
|
||
|
||
## Results
|
||
|
||
Everything lands in `results/` beside the binary: one JSON file per suite and a
|
||
SQLite database accumulating every run, with the model, the endpoint, the
|
||
thinking level, per-item timings, token counts, and the digest of the image that
|
||
produced the score.
|
||
|
||
Two things the tool will refuse to do, both because a wrong number is worse than
|
||
no number: it will not run against an endpoint that is not answering, and it
|
||
will not run infrabench with validators that cannot actually validate.
|