pch/festbench-web
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
festbench-web/README.md

5.6 KiB
Raw Blame History

festbench-web

Web viewer for FestBench results — LLM benchmark datapoints submitted by the festbench CLI, displayed in a fast, sortable table.

Stack: Laravel 12 · Livewire 4 + Volt · Flux UI (Pro) · FrankenPHP · PostgreSQL 17

Architecture (v2)

One table per bench suite, monthly-partitioned on Postgres:

Table Suite-specific columns
softbench_results section, tool_calls
infrabench_results kind (recall/constraint)
prosebench_results section, checks_passed/total, judge, prefill, prompt_tokens
psychbench_results section, checks_passed/total
scenario_runs scenario, outcome, turns, commands, submissions, rejected, t_pass, verdicts

All share the core metric columns (ok, decode, user_tps, ttft, t_answer, wall, reasoning, visible, total, finish) plus a payload JSONB preserving the complete source row — future festbench fields need no migration.

Why: each suite has a different shape; partitioning keeps appends on a hot partition; decommissioning a bench detaches whole partitions; and Postgres inlines the all_results UNION view so the combined home query uses each suite table's indexes directly.

Suite registry — decommissioning & adding benches

config/festbench.php is the single source of truth: suite → model, active flag, in_all_results flag.

  • Decommission a bench: flip active => false, run php artisan festbench:suites:refresh. The suite vanishes from the home table and suite filter; historical data stays queryable and old run-detail links keep rendering.
  • Add a bench: migration + model, registry entry, refresh.

Scale design (built lightly, ready for 10k submissions/day)

  • all_results view — combined home table; suite-filtered queries bypass it and hit the concrete table's indexes.
  • home_snapshot — flat precomputed default view, refreshed every 5 minutes (festbench:snapshot). At high traffic the request path never runs a UNION.
  • TTL staleness caching (60–300s), never event-flush — at high submission rates event flushes would storm the cache; benchmark data a couple of minutes old is correct.
  • Guest pages carry Cache-Control: public, s-maxage=…, stale-while-revalidate so a CDN absorbs the bulk of views. Authenticated views are private, no-store (they embed the user's private runs).
  • Queue-based CLI ingestion + Redis driver: documented as the required shape for the future submit endpoint (auth session is the prerequisite).
  • Retention: everything is kept forever.
  • Monthly partitions exist for ±1 year at migration time; a scheduled partition-creation command is the follow-up when real submission volume approaches the horizon.

Visibility rules

Runs are public by default. Guests see public runs only; authenticated users additionally see their own private runs. Festbench-submitted metrics are immutable — enforced by a model guard and covered by tests; owners can only add notes and toggle visibility.

Local development

Requirements: frankenphp binary on PATH, Node 20+, Docker/Podman.

docker compose -f docker/develop.compose.yaml up -d   # Postgres 17 on 127.0.0.1:5432
./dev.sh                                              # frankenphp :8080 + vite HMR

Composer runs through frankenphp's embedded PHP (the host php.ini disables most extensions):

frankenphp php-cli $(pwd)/vendor/bin/composer install
frankenphp php-cli $(pwd)/vendor/bin/pest
frankenphp php-cli $(pwd)/vendor/bin/pint
frankenphp php-cli artisan <command>

Importing festbench data

# everything from a local benchmarks.db
php artisan festbench:import results/benchmarks.db --user=you@example.com

# one session, surgically, purging existing data first
php artisan festbench:import results/benchmarks.db \
  --user=you@example.com --label="deterministic sampling, repeat 1" --fresh

--label filters by the run's label; --fresh deletes all existing benchmark data first; rows route to per-suite tables via the registry; unknown suites keep their run but warn about skipped rows.

Production (Coolify)

Build pack: Dockerfile. Single FrankenPHP process — no nginx, no php-fpm, no supervisor.

  • 3 stages: composer (Flux Pro auth as a BuildKit secret), node assets, final dunglas/frankenphp:1.12-php8.5 + install-php-extensions pdo_pgsql pgsql intl zip opcache.
  • Entrypoint: migrate --force (retries) → festbench:suites:refresh → festbench:snapshot → frankenphp run.
  • Caddyfile listens on {$SERVER_NAME::8080}; Coolify maps its port.

Secrets/env to set in Coolify:

Variable Notes
APP_KEY generate once, keep stable
DB_HOST / DB_PORT / DB_DATABASE / DB_USERNAME / DB_PASSWORD the nearby Postgres
SESSION_DRIVER=database, CACHE_STORE=database no redis needed yet
FESTBENCH_HOME_CACHE_TTL seconds; default 300

Flux Pro authentication: BuildKit secret composer_auth (contents = the project's auth.json). The license key is never committed.

Health check: GET /up.

Submitting results (CLI integration — separate session)

The schema mirrors festbench's output exactly; festbench:import is the reference implementation of the mapping. The authed ingestion endpoint for festbench --submit is future work (queue-based, Redis-backed at volume).

Notes

  • Scheduler must run in production for festbench:snapshot (Coolify's scheduled task or schedule:work in a sidecar when needed).
  • Worker mode toggle: FRANKENPHP_CONFIG="worker …" — documented option, off by default.
  • Tests run on in-memory SQLite (plain tables); partitioning is Postgres-only.