diff --git a/docs/result-grouping-plan.md b/docs/result-grouping-plan.md new file mode 100644 index 0000000..40b7628 --- /dev/null +++ b/docs/result-grouping-plan.md @@ -0,0 +1,72 @@ +# Result Grouping UI Plan + +## Goal +Allow users to combine compatible benchmark runs after import, including ScenarioBench runs performed in separate batches, without changing the original imported run metadata. + +## Requirements +- Group only runs owned by the same user. +- Require matching visibility, model, quant, confirmed GPU, GPU count, engine/build, server configuration, harness/version/mode, and thinking-level profile. +- Preserve each run's imported label and provenance. +- Use the explicit group as the dashboard/session identity. +- Allow users to remove a run from a group later. + +## ScenarioBench Rules +- Combine selected ScenarioBench runs into one aggregate row. +- Preview all scenarios before confirmation. +- Reject duplicate scenario identities by default. +- When a duplicate exists, require the user to select the retained attempt. +- Do not silently count multiple attempts as separate benchmark scenarios. + +## Regular Suite Rules +- Preview included suites, task counts, thinking levels, pass/fail totals, and duplicate task identities. +- Merge disjoint task batches normally. +- Warn about overlapping tasks and require an explicit decision before grouping. + +## Data Model +1. Add `result_groups`: + - owner/user ID + - display name + - compatibility fingerprint + - visibility + - timestamps +2. Add `result_group_runs`: + - result group ID + - benchmark run ID + - unique membership constraint +3. Keep grouping metadata separate from immutable `benchmark_runs` provenance. + +## Summary Generation +1. Update `festbench:summary`: + - Group runs assigned to a result group by group ID. + - Keep existing metadata-plus-label grouping for ungrouped runs. +2. Display the group name as the session label. +3. Aggregate ScenarioBench only after duplicate-scenario validation. + +## User Interface +1. Add selection controls to **My Runs**. +2. Add actions: + - Create result group + - Add to existing result group + - Remove from result group +3. Show only compatible groups as destinations. +4. Add a confirmation modal with: + - Matching metadata + - Included runs and suites + - Scenario/task overlap warnings + - Aggregate pass/fail preview +5. Rebuild session summaries and the home snapshot after any group membership change. + +## Validation and Authorization +- Enforce ownership on every group and membership operation. +- Block incompatible metadata by default. +- Block duplicate ScenarioBench scenarios by default. +- Block groups crossing visibility boundaries. +- Block grouping runs with different engine builds or server configurations. + +## Tests +1. Two disjoint ScenarioBench batches form one aggregate. +2. Duplicate ScenarioBench scenarios are rejected. +3. Incompatible model, quant, GPU/count, engine/build, config, visibility, harness, or thinking level are rejected. +4. Removing a run restores its ordinary standalone grouping. +5. Users cannot group runs they do not own. +6. Existing label-based grouping remains unchanged for ungrouped runs. \ No newline at end of file