72 lines
2.9 KiB
Markdown
72 lines
2.9 KiB
Markdown
|
# Result Grouping UI Plan
|
||
|
|
|
||
|
|
## Goal
|
||
|
|
Allow users to combine compatible benchmark runs after import, including ScenarioBench runs performed in separate batches, without changing the original imported run metadata.
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
- Group only runs owned by the same user.
|
||
|
|
- Require matching visibility, model, quant, confirmed GPU, GPU count, engine/build, server configuration, harness/version/mode, and thinking-level profile.
|
||
|
|
- Preserve each run's imported label and provenance.
|
||
|
|
- Use the explicit group as the dashboard/session identity.
|
||
|
|
- Allow users to remove a run from a group later.
|
||
|
|
|
||
|
|
## ScenarioBench Rules
|
||
|
|
- Combine selected ScenarioBench runs into one aggregate row.
|
||
|
|
- Preview all scenarios before confirmation.
|
||
|
|
- Reject duplicate scenario identities by default.
|
||
|
|
- When a duplicate exists, require the user to select the retained attempt.
|
||
|
|
- Do not silently count multiple attempts as separate benchmark scenarios.
|
||
|
|
|
||
|
|
## Regular Suite Rules
|
||
|
|
- Preview included suites, task counts, thinking levels, pass/fail totals, and duplicate task identities.
|
||
|
|
- Merge disjoint task batches normally.
|
||
|
|
- Warn about overlapping tasks and require an explicit decision before grouping.
|
||
|
|
|
||
|
|
## Data Model
|
||
|
|
1. Add `result_groups`:
|
||
|
|
- owner/user ID
|
||
|
|
- display name
|
||
|
|
- compatibility fingerprint
|
||
|
|
- visibility
|
||
|
|
- timestamps
|
||
|
|
2. Add `result_group_runs`:
|
||
|
|
- result group ID
|
||
|
|
- benchmark run ID
|
||
|
|
- unique membership constraint
|
||
|
|
3. Keep grouping metadata separate from immutable `benchmark_runs` provenance.
|
||
|
|
|
||
|
|
## Summary Generation
|
||
|
|
1. Update `festbench:summary`:
|
||
|
|
- Group runs assigned to a result group by group ID.
|
||
|
|
- Keep existing metadata-plus-label grouping for ungrouped runs.
|
||
|
|
2. Display the group name as the session label.
|
||
|
|
3. Aggregate ScenarioBench only after duplicate-scenario validation.
|
||
|
|
|
||
|
|
## User Interface
|
||
|
|
1. Add selection controls to **My Runs**.
|
||
|
|
2. Add actions:
|
||
|
|
- Create result group
|
||
|
|
- Add to existing result group
|
||
|
|
- Remove from result group
|
||
|
|
3. Show only compatible groups as destinations.
|
||
|
|
4. Add a confirmation modal with:
|
||
|
|
- Matching metadata
|
||
|
|
- Included runs and suites
|
||
|
|
- Scenario/task overlap warnings
|
||
|
|
- Aggregate pass/fail preview
|
||
|
|
5. Rebuild session summaries and the home snapshot after any group membership change.
|
||
|
|
|
||
|
|
## Validation and Authorization
|
||
|
|
- Enforce ownership on every group and membership operation.
|
||
|
|
- Block incompatible metadata by default.
|
||
|
|
- Block duplicate ScenarioBench scenarios by default.
|
||
|
|
- Block groups crossing visibility boundaries.
|
||
|
|
- Block grouping runs with different engine builds or server configurations.
|
||
|
|
|
||
|
|
## Tests
|
||
|
|
1. Two disjoint ScenarioBench batches form one aggregate.
|
||
|
|
2. Duplicate ScenarioBench scenarios are rejected.
|
||
|
|
3. Incompatible model, quant, GPU/count, engine/build, config, visibility, harness, or thinking level are rejected.
|
||
|
|
4. Removing a run restores its ordinary standalone grouping.
|
||
|
|
5. Users cannot group runs they do not own.
|
||
|
|
6. Existing label-based grouping remains unchanged for ungrouped runs.
|