138 lines
No EOL
5.6 KiB
Markdown
138 lines
No EOL
5.6 KiB
Markdown
# festbench-web
|
||
|
||
Web viewer for [FestBench](../llm_benchmark) results — LLM benchmark datapoints
|
||
submitted by the `festbench` CLI, displayed in a fast, sortable table.
|
||
|
||
Stack: Laravel 12 · Livewire 4 + Volt · Flux UI (Pro) · FrankenPHP · PostgreSQL 17
|
||
|
||
## Architecture (v2)
|
||
|
||
**One table per bench suite**, monthly-partitioned on Postgres:
|
||
|
||
| Table | Suite-specific columns |
|
||
|---|---|
|
||
| `softbench_results` | section, tool_calls |
|
||
| `infrabench_results` | kind (recall/constraint) |
|
||
| `prosebench_results` | section, checks_passed/total, judge, prefill, prompt_tokens |
|
||
| `psychbench_results` | section, checks_passed/total |
|
||
| `scenario_runs` | scenario, outcome, turns, commands, submissions, rejected, t_pass, verdicts |
|
||
|
||
All share the core metric columns (ok, decode, user_tps, ttft, t_answer, wall,
|
||
reasoning, visible, total, finish) plus a `payload JSONB` preserving the
|
||
complete source row — future festbench fields need no migration.
|
||
|
||
**Why:** each suite has a different shape; partitioning keeps appends on a hot
|
||
partition; decommissioning a bench detaches whole partitions; and Postgres
|
||
inlines the `all_results` UNION view so the combined home query uses each
|
||
suite table's indexes directly.
|
||
|
||
### Suite registry — decommissioning & adding benches
|
||
|
||
`config/festbench.php` is the single source of truth: suite → model, `active`
|
||
flag, `in_all_results` flag.
|
||
|
||
- **Decommission a bench**: flip `active => false`, run
|
||
`php artisan festbench:suites:refresh`. The suite vanishes from the home
|
||
table and suite filter; historical data stays queryable and old run-detail
|
||
links keep rendering.
|
||
- **Add a bench**: migration + model, registry entry, refresh.
|
||
|
||
### Scale design (built lightly, ready for 10k submissions/day)
|
||
|
||
- `all_results` view — combined home table; suite-filtered queries bypass it
|
||
and hit the concrete table's indexes.
|
||
- `home_snapshot` — flat precomputed default view, refreshed every 5 minutes
|
||
(`festbench:snapshot`). At high traffic the request path never runs a UNION.
|
||
- TTL staleness caching (60–300s), **never event-flush** — at high submission
|
||
rates event flushes would storm the cache; benchmark data a couple of minutes
|
||
old is correct.
|
||
- Guest pages carry `Cache-Control: public, s-maxage=…, stale-while-revalidate`
|
||
so a CDN absorbs the bulk of views. Authenticated views are `private,
|
||
no-store` (they embed the user's private runs).
|
||
- Queue-based CLI ingestion + Redis driver: documented as the required shape
|
||
for the future submit endpoint (auth session is the prerequisite).
|
||
- Retention: everything is kept forever.
|
||
- Monthly partitions exist for ±1 year at migration time; a scheduled
|
||
partition-creation command is the follow-up when real submission volume
|
||
approaches the horizon.
|
||
|
||
## Visibility rules
|
||
|
||
Runs are public by default. Guests see public runs only; authenticated users
|
||
additionally see their own private runs. Festbench-submitted metrics are
|
||
immutable — enforced by a model guard and covered by tests; owners can only
|
||
add notes and toggle visibility.
|
||
|
||
## Local development
|
||
|
||
Requirements: `frankenphp` binary on PATH, Node 20+, Docker/Podman.
|
||
|
||
```bash
|
||
docker compose -f docker/develop.compose.yaml up -d # Postgres 17 on 127.0.0.1:5432
|
||
./dev.sh # frankenphp :8080 + vite HMR
|
||
```
|
||
|
||
Composer runs through frankenphp's embedded PHP (the host php.ini disables
|
||
most extensions):
|
||
|
||
```bash
|
||
frankenphp php-cli $(pwd)/vendor/bin/composer install
|
||
frankenphp php-cli $(pwd)/vendor/bin/pest
|
||
frankenphp php-cli $(pwd)/vendor/bin/pint
|
||
frankenphp php-cli artisan <command>
|
||
```
|
||
|
||
### Importing festbench data
|
||
|
||
```bash
|
||
# everything from a local benchmarks.db
|
||
php artisan festbench:import results/benchmarks.db --user=you@example.com
|
||
|
||
# one session, surgically, purging existing data first
|
||
php artisan festbench:import results/benchmarks.db \
|
||
--user=you@example.com --label="deterministic sampling, repeat 1" --fresh
|
||
```
|
||
|
||
`--label` filters by the run's label; `--fresh` deletes all existing benchmark
|
||
data first; rows route to per-suite tables via the registry; unknown suites
|
||
keep their run but warn about skipped rows.
|
||
|
||
## Production (Coolify)
|
||
|
||
Build pack: **Dockerfile**. Single FrankenPHP process — no nginx, no php-fpm,
|
||
no supervisor.
|
||
|
||
- 3 stages: composer (Flux Pro auth as a BuildKit **secret**), node assets,
|
||
final `dunglas/frankenphp:1.12-php8.5` + `install-php-extensions pdo_pgsql
|
||
pgsql intl zip opcache`.
|
||
- Entrypoint: `migrate --force` (retries) → `festbench:suites:refresh` →
|
||
`festbench:snapshot` → `frankenphp run`.
|
||
- Caddyfile listens on `{$SERVER_NAME::8080}`; Coolify maps its port.
|
||
|
||
Secrets/env to set in Coolify:
|
||
|
||
| Variable | Notes |
|
||
|---|---|
|
||
| `APP_KEY` | generate once, keep stable |
|
||
| `DB_HOST` / `DB_PORT` / `DB_DATABASE` / `DB_USERNAME` / `DB_PASSWORD` | the nearby Postgres |
|
||
| `SESSION_DRIVER=database`, `CACHE_STORE=database` | no redis needed yet |
|
||
| `FESTBENCH_HOME_CACHE_TTL` | seconds; default 300 |
|
||
|
||
Flux Pro authentication: BuildKit secret `composer_auth` (contents = the
|
||
project's `auth.json`). The license key is never committed.
|
||
|
||
Health check: `GET /up`.
|
||
|
||
## Submitting results (CLI integration — separate session)
|
||
|
||
The schema mirrors festbench's output exactly; `festbench:import` is the
|
||
reference implementation of the mapping. The authed ingestion endpoint for
|
||
`festbench --submit` is future work (queue-based, Redis-backed at volume).
|
||
|
||
## Notes
|
||
|
||
- Scheduler must run in production for `festbench:snapshot` (Coolify's
|
||
scheduled task or `schedule:work` in a sidecar when needed).
|
||
- Worker mode toggle: `FRANKENPHP_CONFIG="worker …"` — documented option, off
|
||
by default.
|
||
- Tests run on in-memory SQLite (plain tables); partitioning is Postgres-only. |