diff --git a/Dockerfile b/Dockerfile index 7f52ea3..1a51096 100644 --- a/Dockerfile +++ b/Dockerfile @@ -27,7 +27,7 @@ RUN npm run build # Stage 3: production image — FrankenPHP serves the app (no nginx, no supervisor) FROM docker.io/dunglas/frankenphp:1.12-php8.5 -RUN install-php-extensions pdo_mysql intl zip opcache +RUN install-php-extensions pdo_pgsql pgsql intl zip opcache WORKDIR /app diff --git a/README.md b/README.md index 6802a6d..5e25506 100644 --- a/README.md +++ b/README.md @@ -3,32 +3,72 @@ # festbench-web Web viewer for [FestBench](../llm_benchmark) results — LLM benchmark datapoints submitted by the `festbench` CLI, displayed in a fast, sortable table. -Stack: Laravel 12 · Livewire 4 + Volt · Flux UI (Pro) · FrankenPHP · MySQL/MariaDB +Stack: Laravel 12 · Livewire 4 + Volt · Flux UI (Pro) · FrankenPHP · PostgreSQL 17 -## Features +## Architecture (v2) -- **Home** — every datapoint festbench captures (per-task rows across all runs), - server-side sortable and filterable (suite, level, model, pass/fail, search). -- **Run detail** (`/runs/{run_id}`) — full provenance: model identity, engine - build, GPU, image digest, validators, suite versions, plus per-task metrics. -- **Ownership overlay** — festbench-submitted data is immutable; owners can add - notes and toggle public/private visibility. Immutability is enforced in the - model (`BenchmarkRun::OVERLAY_FIELDS`) and covered by tests. -- **My Runs** — list your own runs, flip visibility, see pass counts. -- `festbench:import` — import a local festbench `benchmarks.db` (SQLite): - `php artisan festbench:import results/benchmarks.db --user=you@example.com` +**One table per bench suite**, monthly-partitioned on Postgres: + +| Table | Suite-specific columns | +|---|---| +| `softbench_results` | section, tool_calls | +| `infrabench_results` | kind (recall/constraint) | +| `prosebench_results` | section, checks_passed/total, judge, prefill, prompt_tokens | +| `psychbench_results` | section, checks_passed/total | +| `scenario_runs` | scenario, outcome, turns, commands, submissions, rejected, t_pass, verdicts | + +All share the core metric columns (ok, decode, user_tps, ttft, t_answer, wall, +reasoning, visible, total, finish) plus a `payload JSONB` preserving the +complete source row — future festbench fields need no migration. + +**Why:** each suite has a different shape; partitioning keeps appends on a hot +partition; decommissioning a bench detaches whole partitions; and Postgres +inlines the `all_results` UNION view so the combined home query uses each +suite table's indexes directly. + +### Suite registry — decommissioning & adding benches + +`config/festbench.php` is the single source of truth: suite → model, `active` +flag, `in_all_results` flag. + +- **Decommission a bench**: flip `active => false`, run + `php artisan festbench:suites:refresh`. The suite vanishes from the home + table and suite filter; historical data stays queryable and old run-detail + links keep rendering. +- **Add a bench**: migration + model, registry entry, refresh. + +### Scale design (built lightly, ready for 10k submissions/day) + +- `all_results` view — combined home table; suite-filtered queries bypass it + and hit the concrete table's indexes. +- `home_snapshot` — flat precomputed default view, refreshed every 5 minutes + (`festbench:snapshot`). At high traffic the request path never runs a UNION. +- TTL staleness caching (60–300s), **never event-flush** — at high submission + rates event flushes would storm the cache; benchmark data a couple of minutes + old is correct. +- Guest pages carry `Cache-Control: public, s-maxage=…, stale-while-revalidate` + so a CDN absorbs the bulk of views. Authenticated views are `private, + no-store` (they embed the user's private runs). +- Queue-based CLI ingestion + Redis driver: documented as the required shape + for the future submit endpoint (auth session is the prerequisite). +- Retention: everything is kept forever. +- Monthly partitions exist for ±1 year at migration time; a scheduled + partition-creation command is the follow-up when real submission volume + approaches the horizon. ## Visibility rules -Runs are public by default (matching the CLI's redaction philosophy). Guests see -public runs only; authenticated users additionally see their own private runs. +Runs are public by default. Guests see public runs only; authenticated users +additionally see their own private runs. Festbench-submitted metrics are +immutable — enforced by a model guard and covered by tests; owners can only +add notes and toggle visibility. ## Local development Requirements: `frankenphp` binary on PATH, Node 20+, Docker/Podman. ```bash -docker compose -f docker/develop.compose.yaml up -d # MariaDB on 127.0.0.1:3306 +docker compose -f docker/develop.compose.yaml up -d # Postgres 17 on 127.0.0.1:5432 ./dev.sh # frankenphp :8080 + vite HMR ``` @@ -42,49 +82,57 @@ ## Local development frankenphp php-cli artisan ``` -The dev DB deliberately lives in a **container, reached over TCP** — the same -topology as production, where MariaDB runs on a nearby host. Point `.env` -`DB_HOST`/`DB_PORT` at wherever the DB actually lives. +### Importing festbench data + +```bash +# everything from a local benchmarks.db +php artisan festbench:import results/benchmarks.db --user=you@example.com + +# one session, surgically, purging existing data first +php artisan festbench:import results/benchmarks.db \ + --user=you@example.com --label="deterministic sampling, repeat 1" --fresh +``` + +`--label` filters by the run's label; `--fresh` deletes all existing benchmark +data first; rows route to per-suite tables via the registry; unknown suites +keep their run but warn about skipped rows. ## Production (Coolify) -Build pack: **Dockerfile**. The image is a single FrankenPHP process — no -nginx, no php-fpm, no supervisor. +Build pack: **Dockerfile**. Single FrankenPHP process — no nginx, no php-fpm, +no supervisor. -- `Dockerfile` — 3 stages: composer (Flux Pro auth as a BuildKit **secret**), - node assets, final `dunglas/frankenphp:1.12-php8.5` + `install-php-extensions - pdo_mysql intl zip opcache`. -- `docker/entrypoint.sh` — retries `artisan migrate --force`, then `artisan - optimize`, then `exec frankenphp run`. -- `docker/Caddyfile` — listens on `{$SERVER_NAME::8080}`; Coolify maps its - port to it. Static files served directly by Caddy, Laravel via `php_server`. +- 3 stages: composer (Flux Pro auth as a BuildKit **secret**), node assets, + final `dunglas/frankenphp:1.12-php8.5` + `install-php-extensions pdo_pgsql + pgsql intl zip opcache`. +- Entrypoint: `migrate --force` (retries) → `festbench:suites:refresh` → + `festbench:snapshot` → `frankenphp run`. +- Caddyfile listens on `{$SERVER_NAME::8080}`; Coolify maps its port. Secrets/env to set in Coolify: | Variable | Notes | |---|---| | `APP_KEY` | generate once, keep stable | -| `DB_HOST` / `DB_PORT` / `DB_DATABASE` / `DB_USERNAME` / `DB_PASSWORD` | the nearby MariaDB | -| `SESSION_DRIVER=database`, `CACHE_STORE=database` | no redis needed | +| `DB_HOST` / `DB_PORT` / `DB_DATABASE` / `DB_USERNAME` / `DB_PASSWORD` | the nearby Postgres | +| `SESSION_DRIVER=database`, `CACHE_STORE=database` | no redis needed yet | +| `FESTBENCH_HOME_CACHE_TTL` | seconds; default 300 | -Flux Pro authentication: add a secret named `composer_auth` -(file contents = the project's `auth.json`) and configure Coolify to build with -`--secret id=composer_auth,src=…` (or `docker build --secret` in CI). The -license key is never committed. +Flux Pro authentication: BuildKit secret `composer_auth` (contents = the +project's `auth.json`). The license key is never committed. Health check: `GET /up`. ## Submitting results (CLI integration — separate session) -The schema (`benchmark_runs`, `run_results`) mirrors festbench's SQLite exactly, -promotes the sortable columns, and stores the **complete source row as JSON** -(`run_results.payload`) so new festbench fields require no migration. The -authed ingestion endpoint for `festbench --submit` is future work; the -`festbench:import` command is the reference implementation of that mapping. +The schema mirrors festbench's output exactly; `festbench:import` is the +reference implementation of the mapping. The authed ingestion endpoint for +`festbench --submit` is future work (queue-based, Redis-backed at volume). -## Deferred +## Notes -- `scenario_runs` (scenariobench's agentic metrics: turns, submissions, oracle - verdicts) — intentionally not imported yet. -- Worker mode (`FRANKENPHP_CONFIG="worker ..."`) — available as a one-env-var - toggle when throughput demands it. \ No newline at end of file +- Scheduler must run in production for `festbench:snapshot` (Coolify's + scheduled task or `schedule:work` in a sidecar when needed). +- Worker mode toggle: `FRANKENPHP_CONFIG="worker …"` — documented option, off + by default. +- Tests run on in-memory SQLite (plain tables); partitioning is Postgres-only. \ No newline at end of file diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh index 03759c0..f4efa96 100644 --- a/docker/entrypoint.sh +++ b/docker/entrypoint.sh @@ -21,6 +21,8 @@ until php artisan migrate --force --no-ansi; do sleep 3 done +php artisan festbench:suites:refresh --no-ansi || true +php artisan festbench:snapshot --no-ansi || true php artisan optimize --no-ansi || true exec frankenphp run --config /etc/caddy/Caddyfile \ No newline at end of file