festbench: install instructions and suite documentation
Binary releases only. The source is maintained in a private repository.
This commit is contained in:
PeterChrz
commit
28b8e682a8
GPG key ID:
8F0826ECF7302C63
1 changed files with 195 additions and 0 deletions
195
README.md
Normal file
195
README.md
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
# festbench
|
||||
|
||||
An LLM benchmark suite that runs entirely in a container. Point it at any
|
||||
OpenAI-compatible endpoint and it scores writing, code, clinical reasoning,
|
||||
infrastructure-as-code, long-context retrieval, and hands-on repair of a live
|
||||
Kubernetes cluster.
|
||||
|
||||
Nothing is installed on your machine except the binary. No Python, no OpenTofu,
|
||||
no Ansible, no kubectl — only a container runtime.
|
||||
|
||||
This repository distributes the **binary only**. The source is maintained
|
||||
privately.
|
||||
|
||||
---
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.0/festbench_0.3.0_linux_amd64
|
||||
curl -sfLO https://build.diganthrope.com/pch/festbench-llm/releases/download/v0.3.0/SHA256SUMS
|
||||
sha256sum -c SHA256SUMS
|
||||
chmod +x festbench_0.3.0_linux_amd64 && sudo mv festbench_0.3.0_linux_amd64 /usr/local/bin/festbench
|
||||
```
|
||||
|
||||
**Requirements:** Linux x86_64, and Docker or Podman. That is all.
|
||||
|
||||
Check the machine before your first run — it catches the problems that fail
|
||||
confusingly rather than clearly:
|
||||
|
||||
```bash
|
||||
festbench --check
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Flags
|
||||
|
||||
| flag | default | purpose |
|
||||
|---|---|---|
|
||||
| `--endpoint` | `$BENCH_URL` | OpenAI-compatible API base URL. **Required.** |
|
||||
| `--suites` | all one-shot suites | comma-separated: `softbench,prosebench,psychbench,infrabench,depthsweep,scenariobench` |
|
||||
| `--levels` | suite default | thinking levels to sweep: `off,low,medium,xhigh` |
|
||||
| `--repeat` | 1 | samples per item. **Use 3 or more for any number you intend to quote.** |
|
||||
| `--label` | — | free text stored with the run, for telling runs apart later |
|
||||
| `--harness-model` | — | route through a containerised agent, e.g. `github-copilot/claude-sonnet-5:off` |
|
||||
| `--no-harness` | — | talk straight to `--endpoint` (the default behaviour) |
|
||||
| `--scenario` | all | scenariobench only: run one scenario |
|
||||
| `--minutes` | 15 | scenariobench only: per-scenario time budget |
|
||||
| `--image` | `build.diganthrope.com/pch/festbench:latest` | runner image |
|
||||
| `--results` | beside the binary | where the JSON and SQLite database land |
|
||||
| `--runtime` | `auto` | `auto` \| `docker` \| `podman` |
|
||||
| `-i` | — | guided run: checks prerequisites, then asks what to run |
|
||||
|
||||
```bash
|
||||
# simplest thing that works
|
||||
festbench --endpoint https://your-llm:8000 --suites softbench
|
||||
|
||||
# a real measurement, not a single sample
|
||||
festbench --endpoint https://your-llm:8000 --repeat 3 --levels off,xhigh
|
||||
|
||||
# through an agent harness instead of straight to the endpoint
|
||||
festbench --endpoint https://your-llm:8000 --harness-model github-copilot/claude-sonnet-5:off
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The suites
|
||||
|
||||
Six suites. Five send a prompt and score the reply; the sixth puts the model in
|
||||
front of a real broken cluster.
|
||||
|
||||
### softbench — code, prediction, traps
|
||||
|
||||
19 items: write code that must pass tests, predict exactly what a program
|
||||
prints, avoid reasoning traps, and respect forbidden-construct constraints.
|
||||
Output prediction is the discriminating section — the answers are unambiguous
|
||||
and a model either understands the semantics or does not.
|
||||
|
||||
### prosebench — writing against planted ground truth
|
||||
|
||||
10 items scored on adherence and factual control rather than style: word-count
|
||||
limits, required content, forbidden claims. A model that writes beautifully and
|
||||
ignores the brief scores badly, which is the point.
|
||||
|
||||
### psychbench — clinical reasoning
|
||||
|
||||
12 synthetic cases, each built backwards from a known answer with hidden
|
||||
evidence, a decoy diagnosis, and a required action. Scored on whether the model
|
||||
finds the evidence and declines to settle on the decoy.
|
||||
|
||||
### infrabench — infrastructure as code
|
||||
|
||||
10 tasks validated by **real tools**, not regex: OpenTofu, kubectl, kubeconform
|
||||
and ansible-doc all run against the model's output inside the image. This is why
|
||||
the image is pinned — the same manifest gets the same verdict everywhere.
|
||||
Catches things like hallucinated Ansible module names, which pass YAML
|
||||
validation and fail at run time.
|
||||
|
||||
### depthsweep — usable context
|
||||
|
||||
Holds a document at a fixed length and moves one planted fact through it, so the
|
||||
result is a curve rather than a point. The failure that matters is positional:
|
||||
models often retrieve reliably from the start and end of a long context and lose
|
||||
the middle. Filler is generated and frozen, so nothing can be answered from
|
||||
training memory.
|
||||
|
||||
### scenariobench — live cluster repair
|
||||
|
||||
Seven scenarios. Each builds a real two-node Kubernetes cluster, populates it
|
||||
with a fictional company's estate, plants one fault, and gives the model a shell
|
||||
on a jump host with scoped credentials. The model investigates, fixes, and
|
||||
submits; an automated checker inspects the **cluster**, not the model's report,
|
||||
and answers only accepted or rejected.
|
||||
|
||||
Faults include RBAC denials, unschedulable workloads, expired certificates, DNS
|
||||
policy mismatches, an OOM kill whose cause is a different workload, a broken
|
||||
kubelet runtime endpoint, and a systemd unit failing at boot.
|
||||
|
||||
---
|
||||
|
||||
## Options that change what you measure
|
||||
|
||||
### Thinking levels
|
||||
|
||||
`--levels off,low,medium,xhigh` sweeps the model's reasoning effort. The cost is
|
||||
not linear and is worth knowing before you start — measured on Qwen3.8-27B:
|
||||
|
||||
| suite | `off` | `low` | `medium` | `xhigh` |
|
||||
|---|---|---|---|---|
|
||||
| softbench | 1.2s | 4.5s | 4.4s | 6.1s |
|
||||
| infrabench | 2.6s | 12.4s | 18.7s | **97.1s** |
|
||||
| prosebench | 7.1s | 17.0s | 23.7s | 39.6s |
|
||||
| psychbench | 11.7s | 32.5s | 55.1s | **117.0s** |
|
||||
|
||||
*(seconds per item)*
|
||||
|
||||
`xhigh` can cost **37×** `off` on the same tasks. Whether it buys anything is
|
||||
the question the sweep answers — on some suites it gains a point, on others
|
||||
nothing at all.
|
||||
|
||||
### Harness on or off
|
||||
|
||||
**Off (default)** — the suite talks straight to `--endpoint`. This measures the
|
||||
model.
|
||||
|
||||
**On** — `--harness-model provider/model` routes every request through a
|
||||
containerised pi.dev agent with its own tools. This measures the model *in an
|
||||
agent loop*, which is closer to how it would actually be used, and lets a run
|
||||
reach any provider the harness is logged in to (GitHub Copilot, Anthropic,
|
||||
OpenAI) rather than only your endpoint.
|
||||
|
||||
Thinking levels work differently in each mode. Against a local endpoint the
|
||||
level travels in the chat template and is sweepable. Through the harness it
|
||||
rides on the model name — `claude-sonnet-5:high` — and a run without a suffix is
|
||||
recorded as `default` rather than claiming a level nobody set.
|
||||
|
||||
---
|
||||
|
||||
## How long it takes
|
||||
|
||||
Measured, not estimated. Wall-clock for a complete run of each suite at
|
||||
`--repeat 1`:
|
||||
|
||||
| suite | items | `off` | `xhigh` |
|
||||
|---|---|---|---|
|
||||
| softbench | 19 | **2 min** | 4 min |
|
||||
| infrabench | 10 | **3 min** | 24 min |
|
||||
| prosebench | 10 | **6 min** | 7 min |
|
||||
| psychbench | 12 | **12 min** | 23 min |
|
||||
| depthsweep | 10 | **13 min** | 13 min |
|
||||
| all five | 61 | **~35 min** | ~70 min |
|
||||
|
||||
**scenariobench is different.** It is time-boxed rather than item-counted: a
|
||||
scenario runs until the model submits an accepted fix or the budget expires.
|
||||
Across 39 runs the mean was **9 minutes** per scenario (range 1–23), plus 2–3
|
||||
minutes to build and tear down the cluster. All seven scenarios at the default
|
||||
15-minute budget is a **1–2 hour** job.
|
||||
|
||||
Timings scale with `--repeat`, and depthsweep scales with context size — the
|
||||
figures above are at its default, and a 100K-token run costs roughly 3 minutes
|
||||
per item because the whole context must be processed before a single token comes
|
||||
back.
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
Everything lands in `results/` beside the binary: one JSON file per suite and a
|
||||
SQLite database accumulating every run, with the model, the endpoint, the
|
||||
thinking level, per-item timings, token counts, and the digest of the image that
|
||||
produced the score.
|
||||
|
||||
Two things the tool will refuse to do, both because a wrong number is worse than
|
||||
no number: it will not run against an endpoint that is not answering, and it
|
||||
will not run infrabench with validators that cannot actually validate.
|
||||
Loading…
Reference in a new issue