festbench-web/README.md

# festbench-web

Web viewer for [FestBench](../llm_benchmark) results — LLM benchmark datapoints
submitted by the `festbench` CLI, displayed in a fast, sortable table.

Stack: Laravel 12 · Livewire 4 + Volt · Flux UI (Pro) · FrankenPHP · PostgreSQL 17

## Architecture (v2)

**One table per bench suite**, monthly-partitioned on Postgres:

| Table | Suite-specific columns |
|---|---|
| `softbench_results` | section, tool_calls |
| `infrabench_results` | kind (recall/constraint) |
| `prosebench_results` | section, checks_passed/total, judge, prefill, prompt_tokens |
| `psychbench_results` | section, checks_passed/total |
| `scenario_runs` | scenario, outcome, turns, commands, submissions, rejected, t_pass, verdicts |

All share the core metric columns (ok, decode, user_tps, ttft, t_answer, wall,
reasoning, visible, total, finish) plus a `payload JSONB` preserving the
complete source row — future festbench fields need no migration.

**Why:** each suite has a different shape; partitioning keeps appends on a hot
partition; decommissioning a bench detaches whole partitions; and Postgres
inlines the `all_results` UNION view so the combined home query uses each
suite table's indexes directly.

### Suite registry — decommissioning & adding benches

`config/festbench.php` is the single source of truth: suite → model, `active`
flag, `in_all_results` flag.

- **Decommission a bench**: flip `active => false`, run
  `php artisan festbench:suites:refresh`. The suite vanishes from the home
  table and suite filter; historical data stays queryable and old run-detail
  links keep rendering.
- **Add a bench**: migration + model, registry entry, refresh.

### Scale design (built lightly, ready for 10k submissions/day)

- `all_results` view — combined home table; suite-filtered queries bypass it
  and hit the concrete table's indexes.
- `home_snapshot` — flat precomputed default view, refreshed every 5 minutes
  (`festbench:snapshot`). At high traffic the request path never runs a UNION.
- TTL staleness caching (60–300s), **never event-flush** — at high submission
  rates event flushes would storm the cache; benchmark data a couple of minutes
  old is correct.
- Guest pages carry `Cache-Control: public, s-maxage=…, stale-while-revalidate`
  so a CDN absorbs the bulk of views. Authenticated views are `private,
  no-store` (they embed the user's private runs).
- Queue-based CLI ingestion + Redis driver: documented as the required shape
  for the future submit endpoint (auth session is the prerequisite).
- Retention: everything is kept forever.
- Monthly partitions exist for ±1 year at migration time; a scheduled
  partition-creation command is the follow-up when real submission volume
  approaches the horizon.

## Visibility rules

Runs are public by default. Guests see public runs only; authenticated users
additionally see their own private runs. Festbench-submitted metrics are
immutable — enforced by a model guard and covered by tests; owners can only
add notes and toggle visibility.

## Local development

Requirements: `frankenphp` binary on PATH, Node 20+, Docker/Podman.

```bash
docker compose -f docker/develop.compose.yaml up -d   # Postgres 17 on 127.0.0.1:5432
./dev.sh                                              # frankenphp :8080 + vite HMR
```

Composer runs through frankenphp's embedded PHP (the host php.ini disables
most extensions):

```bash
frankenphp php-cli $(pwd)/vendor/bin/composer install
frankenphp php-cli $(pwd)/vendor/bin/pest
frankenphp php-cli $(pwd)/vendor/bin/pint
frankenphp php-cli artisan <command>
```

### Importing festbench data

```bash
# everything from a local benchmarks.db
php artisan festbench:import results/benchmarks.db --user=you@example.com

# one session, surgically, purging existing data first
php artisan festbench:import results/benchmarks.db \
  --user=you@example.com --label="deterministic sampling, repeat 1" --fresh
```

`--label` filters by the run's label; `--fresh` deletes all existing benchmark
data first; rows route to per-suite tables via the registry; unknown suites
keep their run but warn about skipped rows.

## Production (Coolify)

Build pack: **Dockerfile**. Single FrankenPHP process — no nginx, no php-fpm,
no supervisor.

- 3 stages: composer (Flux Pro auth as a BuildKit **secret**), node assets,
  final `dunglas/frankenphp:1.12-php8.5` + `install-php-extensions pdo_pgsql
  pgsql intl zip opcache`.
- Entrypoint: `migrate --force` (retries) → `festbench:suites:refresh` →
  `festbench:snapshot` → `frankenphp run`.
- Caddyfile listens on `{$SERVER_NAME::8080}`; Coolify maps its port.

Secrets/env to set in Coolify:

| Variable | Notes |
|---|---|
| `APP_KEY` | generate once, keep stable |
| `DB_HOST` / `DB_PORT` / `DB_DATABASE` / `DB_USERNAME` / `DB_PASSWORD` | the nearby Postgres |
| `SESSION_DRIVER=database`, `CACHE_STORE=database` | no redis needed yet |
| `FESTBENCH_HOME_CACHE_TTL` | seconds; default 300 |

Flux Pro authentication: BuildKit secret `composer_auth` (contents = the
project's `auth.json`). The license key is never committed.

Health check: `GET /up`.

## Submitting results (CLI integration — separate session)

The schema mirrors festbench's output exactly; `festbench:import` is the
reference implementation of the mapping. The authed ingestion endpoint for
`festbench --submit` is future work (queue-based, Redis-backed at volume).

## Notes

- Scheduler must run in production for `festbench:snapshot` (Coolify's
  scheduled task or `schedule:work` in a sidecar when needed).
- Worker mode toggle: `FRANKENPHP_CONFIG="worker …"` — documented option, off
  by default.
- Tests run on in-memory SQLite (plain tables); partitioning is Postgres-only.