feat(seahorse): implement short-term memory engine (LCM) (#2285)
* feat(seahorse): implement short-term memory engine of seahorse
Add pkg/seahorse/ module implementing a SQLite-backed DAG-based summary
hierarchy for context management, ported from lossless-claw's LCM design:
- types.go + short_constants.go: core types (Message, Summary, Conversation,
ContextItem) and configuration constants (fanout, token targets, thresholds)
- migration.go: idempotent DB schema with FTS5 trigram tokenizer for CJK
- store.go: full SQLite CRUD (conversations, messages, summaries DAG,
context_items with ordinal gap numbering, FTS5 search)
- short_engine.go: Engine lifecycle (NewEngine, Ingest, Assemble, Compact),
session pattern filtering (ignore/stateless glob→regex compilation),
per-session mutex via sync.Map
- short_assembler.go: budget-aware context assembly with fresh tail protection
(32 messages), oldest-first eviction, summary XML formatting, RebuildContextItems
- short_compaction.go: leaf compaction (messages→summary) and condensed
compaction (summaries→higher-level summary), 3-level LLM escalation,
CompactUntilUnder for emergency overflow
- short_retrieval.go: lookupByID, FTS5/LIKE search, recursive expand with
token cap
- context_seahorse.go: agent.ContextManager adapter, registered as "seahorse",
provider↔seahorse message type conversion (ToolCalls, tool_result)
* fix(seahorse): correct 3 adapter bugs in context management
- TokenCount: use full message (Content+ToolCalls+Media) instead of Content-only
- Empty Content: rebuild Content from tool_result Parts when stored empty
- Duplicate summaries: summaries only in Summary field, not in History messages
- Grep: fix SearchResult.Snippet→Content for summaries
- Schema: fix FTS5 SQL uses VIRTUAL TABLE not TEMP TABLE
- TestFTS5SQLConstants: verify FTS5 SQL syntax correctness
- Test: fix flaky TestCompactLeaf
* fix(agent): ingest steering messages into seahorse SQLite
Steering messages were only persisted to session JSONL but not ingested
into seahorse SQLite, causing them to be missing from context assembly.
Added `ts.ingestMessage(turnCtx, al, pm)` call in the steering message
injection block alongside the existing JSONL persistence.
Test: TestSeahorseSteeringMessageIngested verifies steering messages
appear in seahorse SQLite DB after being processed.
* fix(seahorse): address 3 blocking bugs from code review
- Fix resequenceContextItemsTx scan error handling (store.go:850)
Changed `return err` to `return scanErr` to properly propagate scan errors
instead of returning nil (which silently corrupts data)
- Fix sql.NullString for INTEGER column (store.go:847)
Changed `mid` from sql.NullString to sql.NullInt64 since message_id
is INTEGER in schema. Removed unnecessary strconv.ParseInt call.
- Fix compactCondensed fallback deleting non-candidate items
Added ReplaceContextItemsWithSummary method for per-item deletion
when candidates are not contiguous in ordinal space.
Optimized to use range deletion when candidates are consecutive.
* fix(seahorse): pass Budget to Compact for correct condensed threshold
Issue #4 from PR review: When Budget was not passed to seahorse.Compact,
it defaulted to `tokensBefore * 0.75`, making `tokensBefore > budget`
always true and causing condensed compaction to trigger unnecessarily.
Changes:
- context_seahorse.go: Forward Budget from CompactRequest to CompactInput
- loop.go: Pass Budget (ContextWindow) in all 3 Compact calls
- Add test verifying condensed is skipped when tokens < threshold
- Fix lint issues in store.go and store_test.go
* fix(seahorse): add mutex for assembler lazy initialization
Issue #5 from PR review: The check-then-create pattern for e.assembler
was a data race when multiple goroutines called Assemble() concurrently:
if e.assembler == nil {
e.assembler = &Assembler{...}
}
Changes:
- Add assemblerMu sync.Mutex to Engine struct
- Add initAssemblerOnce() using double-checked locking (same pattern as initCompactionOnce)
- Add TestAssemblerLazyInitRace to verify thread-safety
* fix(seahorse): handle non-consecutive depths in selectShallowestCondensationCandidate
Issue #8 from PR review: the loop iterated depth 0, 1, 2... assuming
consecutive keys, but break when key was missing caused deeper depths
to never be checked.
Fix: collect all existing depth keys, sort, then iterate in order.
* fix(seahorse): wrap DeleteMessagesAfterID and appendContextItems in transactions
- DeleteMessagesAfterID: wrap all DELETE operations in a transaction for
atomicity, remove redundant manual FTS delete (handled by trigger)
- appendContextItems: use transaction to fix read-then-write race condition
- Add GetMaxOrdinalTx and resolveItemTokenCountTx for transaction-scoped queries
- Remove unused resolveItemTokenCount function
Fixes PR review issues 6 and 7.
* fix(seahorse): derive readable content from Parts and cap CompactUntilUnder iterations
- Derive readable content from MessageParts in AddMessageWithParts so
FTS5 indexing and summary formatting can access tool call information
- formatMessagesForSummary and truncateSummary now fall back to Parts
when Content is empty, fixing blank summaries for Part-based messages
- Add MaxCompactIterations (20) to prevent CompactUntilUnder infinite
loops; exceeded iterations are logged as warnings
2026-04-05 01:05:16 +00:00
|
|
|
package seahorse
|
|
|
|
|
|
|
|
|
|
import (
|
|
|
|
|
"context"
|
|
|
|
|
"encoding/json"
|
|
|
|
|
"fmt"
|
|
|
|
|
|
|
|
|
|
"github.com/sipeed/picoclaw/pkg/tools"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
// ExpandTool recovers full message content by ID.
|
|
|
|
|
type ExpandTool struct {
|
|
|
|
|
engine *RetrievalEngine
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func NewExpandTool(engine *RetrievalEngine) *ExpandTool {
|
|
|
|
|
return &ExpandTool{engine: engine}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *ExpandTool) Name() string {
|
|
|
|
|
return "short_expand"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *ExpandTool) Description() string {
|
|
|
|
|
return `Get full message content by ID.
|
|
|
|
|
|
|
|
|
|
Use when short_grep returns messages and you need complete content (not just snippet).
|
|
|
|
|
|
|
|
|
|
Parameters:
|
|
|
|
|
- message_ids (required): Array of message ID strings (from short_grep results)
|
|
|
|
|
|
|
|
|
|
Returns message with:
|
|
|
|
|
- content: Full text content
|
|
|
|
|
- parts: Structured content
|
|
|
|
|
- text: Full text
|
|
|
|
|
- tool_use: name, arguments, toolCallId
|
|
|
|
|
- tool_result: toolCallId only (content omitted - re-run tool if needed)
|
|
|
|
|
- media: mediaUri (file path), mimeType
|
|
|
|
|
|
|
|
|
|
Notes:
|
|
|
|
|
- tool_result content is not returned (can be large). Re-run the tool if you need the result.
|
|
|
|
|
- Media files are stored on disk at mediaUri path, use bash to access.
|
|
|
|
|
|
|
|
|
|
Example:
|
|
|
|
|
{"message_ids": ["10", "25"]}`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *ExpandTool) Parameters() map[string]any {
|
|
|
|
|
return map[string]any{
|
|
|
|
|
"type": "object",
|
|
|
|
|
"properties": map[string]any{
|
|
|
|
|
"message_ids": map[string]any{
|
|
|
|
|
"type": "array",
|
|
|
|
|
"items": map[string]any{"type": "string"},
|
|
|
|
|
"description": "Message IDs to expand (from short_grep results, e.g., [\"10\", \"25\"])",
|
|
|
|
|
},
|
|
|
|
|
},
|
|
|
|
|
"required": []string{"message_ids"},
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (t *ExpandTool) Execute(ctx context.Context, args map[string]any) *tools.ToolResult {
|
|
|
|
|
idsRaw, ok := args["message_ids"].([]any)
|
|
|
|
|
if !ok || len(idsRaw) == 0 {
|
|
|
|
|
return tools.ErrorResult(
|
|
|
|
|
"Missing required 'message_ids' argument. " +
|
|
|
|
|
"Example: {\"message_ids\": [\"10\", \"25\"]}")
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Parse message IDs
|
|
|
|
|
messageIDs := make([]int64, 0, len(idsRaw))
|
|
|
|
|
for _, id := range idsRaw {
|
|
|
|
|
switch v := id.(type) {
|
|
|
|
|
case string:
|
|
|
|
|
var n int64
|
|
|
|
|
if _, err := fmt.Sscanf(v, "%d", &n); err != nil {
|
|
|
|
|
return tools.ErrorResult(fmt.Sprintf("Invalid message_id %q: %v", v, err))
|
|
|
|
|
}
|
|
|
|
|
messageIDs = append(messageIDs, n)
|
|
|
|
|
case float64:
|
|
|
|
|
messageIDs = append(messageIDs, int64(v))
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
result, err := t.engine.ExpandMessages(ctx, messageIDs)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return tools.ErrorResult("Expand failed: " + err.Error())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Build response with filtered parts
|
|
|
|
|
messages := make([]map[string]any, 0, len(result.Messages))
|
|
|
|
|
for _, msg := range result.Messages {
|
|
|
|
|
parts := make([]map[string]any, 0, len(msg.Parts))
|
|
|
|
|
for _, p := range msg.Parts {
|
|
|
|
|
part := map[string]any{"type": p.Type}
|
|
|
|
|
switch p.Type {
|
|
|
|
|
case "text":
|
|
|
|
|
part["text"] = p.Text
|
|
|
|
|
case "tool_use":
|
|
|
|
|
part["name"] = p.Name
|
|
|
|
|
part["arguments"] = p.Arguments
|
|
|
|
|
part["toolCallId"] = p.ToolCallID
|
|
|
|
|
case "tool_result":
|
|
|
|
|
// Omit content - can be large, re-run tool if needed
|
|
|
|
|
part["toolCallId"] = p.ToolCallID
|
|
|
|
|
case "media":
|
|
|
|
|
part["mediaUri"] = p.MediaURI
|
|
|
|
|
part["mimeType"] = p.MimeType
|
|
|
|
|
}
|
|
|
|
|
parts = append(parts, part)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
messages = append(messages, map[string]any{
|
|
|
|
|
"id": fmt.Sprintf("%d", msg.ID),
|
|
|
|
|
"role": msg.Role,
|
|
|
|
|
"content": msg.Content,
|
|
|
|
|
"parts": parts,
|
|
|
|
|
"conversationId": msg.ConversationID,
|
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
output := map[string]any{
|
|
|
|
|
"success": true,
|
|
|
|
|
"tokenCount": result.TokenCount,
|
|
|
|
|
"messages": messages,
|
|
|
|
|
}
|
2026-06-15 12:31:34 +00:00
|
|
|
data, err := json.Marshal(output)
|
|
|
|
|
if err != nil {
|
|
|
|
|
return tools.ErrorResult(fmt.Sprintf("failed to marshal expand result: %v", err))
|
|
|
|
|
}
|
feat(seahorse): implement short-term memory engine (LCM) (#2285)
* feat(seahorse): implement short-term memory engine of seahorse
Add pkg/seahorse/ module implementing a SQLite-backed DAG-based summary
hierarchy for context management, ported from lossless-claw's LCM design:
- types.go + short_constants.go: core types (Message, Summary, Conversation,
ContextItem) and configuration constants (fanout, token targets, thresholds)
- migration.go: idempotent DB schema with FTS5 trigram tokenizer for CJK
- store.go: full SQLite CRUD (conversations, messages, summaries DAG,
context_items with ordinal gap numbering, FTS5 search)
- short_engine.go: Engine lifecycle (NewEngine, Ingest, Assemble, Compact),
session pattern filtering (ignore/stateless glob→regex compilation),
per-session mutex via sync.Map
- short_assembler.go: budget-aware context assembly with fresh tail protection
(32 messages), oldest-first eviction, summary XML formatting, RebuildContextItems
- short_compaction.go: leaf compaction (messages→summary) and condensed
compaction (summaries→higher-level summary), 3-level LLM escalation,
CompactUntilUnder for emergency overflow
- short_retrieval.go: lookupByID, FTS5/LIKE search, recursive expand with
token cap
- context_seahorse.go: agent.ContextManager adapter, registered as "seahorse",
provider↔seahorse message type conversion (ToolCalls, tool_result)
* fix(seahorse): correct 3 adapter bugs in context management
- TokenCount: use full message (Content+ToolCalls+Media) instead of Content-only
- Empty Content: rebuild Content from tool_result Parts when stored empty
- Duplicate summaries: summaries only in Summary field, not in History messages
- Grep: fix SearchResult.Snippet→Content for summaries
- Schema: fix FTS5 SQL uses VIRTUAL TABLE not TEMP TABLE
- TestFTS5SQLConstants: verify FTS5 SQL syntax correctness
- Test: fix flaky TestCompactLeaf
* fix(agent): ingest steering messages into seahorse SQLite
Steering messages were only persisted to session JSONL but not ingested
into seahorse SQLite, causing them to be missing from context assembly.
Added `ts.ingestMessage(turnCtx, al, pm)` call in the steering message
injection block alongside the existing JSONL persistence.
Test: TestSeahorseSteeringMessageIngested verifies steering messages
appear in seahorse SQLite DB after being processed.
* fix(seahorse): address 3 blocking bugs from code review
- Fix resequenceContextItemsTx scan error handling (store.go:850)
Changed `return err` to `return scanErr` to properly propagate scan errors
instead of returning nil (which silently corrupts data)
- Fix sql.NullString for INTEGER column (store.go:847)
Changed `mid` from sql.NullString to sql.NullInt64 since message_id
is INTEGER in schema. Removed unnecessary strconv.ParseInt call.
- Fix compactCondensed fallback deleting non-candidate items
Added ReplaceContextItemsWithSummary method for per-item deletion
when candidates are not contiguous in ordinal space.
Optimized to use range deletion when candidates are consecutive.
* fix(seahorse): pass Budget to Compact for correct condensed threshold
Issue #4 from PR review: When Budget was not passed to seahorse.Compact,
it defaulted to `tokensBefore * 0.75`, making `tokensBefore > budget`
always true and causing condensed compaction to trigger unnecessarily.
Changes:
- context_seahorse.go: Forward Budget from CompactRequest to CompactInput
- loop.go: Pass Budget (ContextWindow) in all 3 Compact calls
- Add test verifying condensed is skipped when tokens < threshold
- Fix lint issues in store.go and store_test.go
* fix(seahorse): add mutex for assembler lazy initialization
Issue #5 from PR review: The check-then-create pattern for e.assembler
was a data race when multiple goroutines called Assemble() concurrently:
if e.assembler == nil {
e.assembler = &Assembler{...}
}
Changes:
- Add assemblerMu sync.Mutex to Engine struct
- Add initAssemblerOnce() using double-checked locking (same pattern as initCompactionOnce)
- Add TestAssemblerLazyInitRace to verify thread-safety
* fix(seahorse): handle non-consecutive depths in selectShallowestCondensationCandidate
Issue #8 from PR review: the loop iterated depth 0, 1, 2... assuming
consecutive keys, but break when key was missing caused deeper depths
to never be checked.
Fix: collect all existing depth keys, sort, then iterate in order.
* fix(seahorse): wrap DeleteMessagesAfterID and appendContextItems in transactions
- DeleteMessagesAfterID: wrap all DELETE operations in a transaction for
atomicity, remove redundant manual FTS delete (handled by trigger)
- appendContextItems: use transaction to fix read-then-write race condition
- Add GetMaxOrdinalTx and resolveItemTokenCountTx for transaction-scoped queries
- Remove unused resolveItemTokenCount function
Fixes PR review issues 6 and 7.
* fix(seahorse): derive readable content from Parts and cap CompactUntilUnder iterations
- Derive readable content from MessageParts in AddMessageWithParts so
FTS5 indexing and summary formatting can access tool call information
- formatMessagesForSummary and truncateSummary now fall back to Parts
when Content is empty, fixing blank summaries for Part-based messages
- Add MaxCompactIterations (20) to prevent CompactUntilUnder infinite
loops; exceeded iterations are logged as warnings
2026-04-05 01:05:16 +00:00
|
|
|
return tools.NewToolResult(string(data))
|
|
|
|
|
}
|