festbench-web/docs/result-grouping-plan.md

Result Grouping UI Plan

Goal

Allow users to combine compatible benchmark runs after import, including ScenarioBench runs performed in separate batches, without changing the original imported run metadata.

Requirements

• Group only runs owned by the same user.
• Require matching visibility, model, quant, confirmed GPU, GPU count, engine/build, server configuration, harness/version/mode, and thinking-level profile.
• Preserve each run's imported label and provenance.
• Use the explicit group as the dashboard/session identity.
• Allow users to remove a run from a group later.

ScenarioBench Rules

• Combine selected ScenarioBench runs into one aggregate row.
• Preview all scenarios before confirmation.
• Reject duplicate scenario identities by default.
• When a duplicate exists, require the user to select the retained attempt.
• Do not silently count multiple attempts as separate benchmark scenarios.

Regular Suite Rules

• Preview included suites, task counts, thinking levels, pass/fail totals, and duplicate task identities.
• Merge disjoint task batches normally.
• Warn about overlapping tasks and require an explicit decision before grouping.

Data Model

1. Add result_groups:
   • owner/user ID
   • display name
   • compatibility fingerprint
   • visibility
   • timestamps
2. Add result_group_runs:
   • result group ID
   • benchmark run ID
   • unique membership constraint
3. Keep grouping metadata separate from immutable benchmark_runs provenance.

Summary Generation

1. Update festbench:summary:
   • Group runs assigned to a result group by group ID.
   • Keep existing metadata-plus-label grouping for ungrouped runs.
2. Display the group name as the session label.
3. Aggregate ScenarioBench only after duplicate-scenario validation.

User Interface

1. Add selection controls to My Runs.
2. Add actions:
   • Create result group
   • Add to existing result group
   • Remove from result group
3. Show only compatible groups as destinations.
4. Add a confirmation modal with:
   • Matching metadata
   • Included runs and suites
   • Scenario/task overlap warnings
   • Aggregate pass/fail preview
5. Rebuild session summaries and the home snapshot after any group membership change.

Validation and Authorization

• Enforce ownership on every group and membership operation.
• Block incompatible metadata by default.
• Block duplicate ScenarioBench scenarios by default.
• Block groups crossing visibility boundaries.
• Block grouping runs with different engine builds or server configurations.

Tests

1. Two disjoint ScenarioBench batches form one aggregate.
2. Duplicate ScenarioBench scenarios are rejected.
3. Incompatible model, quant, GPU/count, engine/build, config, visibility, harness, or thinking level are rejected.
4. Removing a run restores its ordinary standalone grouping.
5. Users cannot group runs they do not own.
6. Existing label-based grouping remains unchanged for ungrouped runs.