zubble/DESIGN.md

# zubble — A Minimal Agentic Harness

> A hackable, extensible agent loop that combines the simplicity of
> [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) with the
> anti-loop discipline of [Ralph Wiggum](https://github.com/fstandhartinger/ralph-wiggum),
> the doom-loop detection of [ZeroStack](https://github.com/gi-dellav/zerostack),
> and hard-won lessons from FestBench's ScenarioBench harness.

---

## Design Principles

1. **Readable.** The core agent loop is ~200 lines of Python. A new
   contributor should understand the entire control flow in one sitting.
   No magic, no metaprogramming, no framework tax.

2. **Extensible.** Three pluggable interfaces — `Model`, `Environment`,
   `Observer` — cover every extension point. New providers, sandboxes, or
   grading hooks are subclasses, not config gymnastics.

3. **Sharable.** MIT licensed. No FestBench dependency. Runs standalone
   against any OpenAI-compatible endpoint, hosted API, or local CLI.
   Pip-installable. Single-file deployable.

4. **Pythonic.** Type-hinted, dataclass configs, stdlib-first. Pydantic
   for config validation (same as mini-swe-agent). No Rust, no Go, no
   Node — the language the research community already speaks.

5. **Honest.** Every failure mode is recorded, not hidden. Degenerate
   loops are detected and logged, not silently retried. Token costs are
   tracked per call, not estimated after the fact.

---

## Why "zubble"?

Named after the world's best dog. A good agentic harness is Man's best
friend — loyal, dependable, always ready to help you fix what's broken.

---

## Origin: What We Learned from FestBench

FestBench's `scenariobench.py` gave a model a broken Kubernetes cluster,
a wall-clock budget, and a free-form text protocol. At xhigh thinking
levels, the model generated 79K characters of degenerate conversation
simulation — emitting both `User:` and `Assistant:` turns in a single
response, 7,303 repetitions of the word "truncated" before timing out.

Five root causes were identified:

1. **No stop sequences** — the API call had no `stop=["User:"]`, so
   generation continued through simulated conversation turns.
2. **Flat-text rendering** — the harness flattened the message history
   into a single string with `User:`/`Assistant:` markers, then sent it
   as one user message. The model saw the pattern and copied it.
3. **No in-stream repetition detection** — the livelock detector only
   checked extracted commands between turns, not repeated text within
   a single generation.
4. **Unbounded maxtok** — xhigh's 32K token budget at 41.5 tok/s = 770s,
   longer than the 480s timeout. One degenerate turn ate the entire
   scenario budget.
5. **No context reset** — when the model got stuck, the accumulated
   context (including the 79K-char degenerate response) poisoned every
   subsequent turn.

`zubble` is designed to make all five of these failure modes structurally
impossible.

---

## Architecture

```
stream_chat()              ← shared HTTP client (unchanged, no ledger awareness)
    ↑
    ├── softbench.py           → calls stream_chat once per task, scores output
    ├── infrabench.py          → calls stream_chat once per task, validates output
    ├── prosebench.py          → calls stream_chat once per task, scores output
    ├── psychbench.py          → calls stream_chat once per task, scores output
    └── scenariobench.py       → calls agent_loop() which calls stream_chat N times
                                    ↑
                              agent_loop (zubble)
                                  ├── protocol.py     ← JSON schema, system prompt, parse-retry
                                  ├── ledger.py       ← plan state machine (short-term memory)
                                  └── repetition.py   ← in-stream repetition detection
```

The plan-state ledger is the model's short-term memory. It tracks not
just what was attempted, but the full causal chain: what the model
planned to do, which steps are done vs pending, where execution diverged
from the plan, and why previous approaches were abandoned.

### Protocol

The model communicates in structured JSON, not free-form text:

```json
{
  "action": "plan" | "run" | "report" | "submit" | "ask",
  "command": "<shell command, for action=run>",
  "plan_step_id": <int, references the plan>,
  "reasoning": "<brief explanation>",
  "steps": ["<step 1>", "<step 2>", ...],  // for action=plan only
  "report": "<short submission summary>",   // for action=report/submit only
  "question": "<clarification>",            // for action=ask only
  "revised_plan": ["<new steps>"]           // when abandoning current plan
}
```

Turn 1: model emits `action: "plan"` with an ordered list of steps.
Turns 2..N: model emits `action: "run"` with `plan_step_id` referencing
which step this action fulfills. The harness checks each action against
the plan state machine. If the action doesn't match the next pending
step, a divergence warning is injected before the command executes.

`submit` is the only verification path. A rejected submission tells the
model to keep working and submit again; the harness owns termination through
the deadline, turn cap, or doom-loop safeguards. The full post-mortem belongs
in the scenario report file, while the JSON `report` field is a short,
single-line summary.

JSON enforcement combines endpoint `response_format: json_schema`, strict
single-object parsing, and one targeted repair generation on malformed output.
After a failed repair, the scenario exits as `parse_error` with the parser
reason rather than spending its remaining budget in an invalid-JSON loop.

### Plan-state ledger

```
PLAN (captured at turn 1):
  Step 1: Diagnose bad cert           → DONE (found expired cert on node-b)
  Step 2: Create new cert manifest    → DONE (cert-fix.yaml created)
  Step 3: Apply manifest              → PENDING
  Step 4: Verify cert renewed         → PENDING

ABANDONED APPROACHES (with causal failure reasons):
  - "restart kubelet" → failed because kubelet was already running;
    cert expiry was root cause, not service state

CURRENT STATE:
  Next expected step: Step 3 (apply manifest)
  Last action: kubectl create -f cert-fix.yaml (step 2, succeeded)
  Divergence flag: NONE
```

Injected into every prompt as "## SESSION MEMORY" so the model sees its
plan, progress, and abandoned approaches before choosing its next action.

### Repetition detection (Phase 2)

Rolling 10-gram CRC32 fingerprint over the last 200 generated tokens.
Graduated thresholds prevent false-positive kills of valid long-running
processes:

| Threshold | Action | Rationale |
|-----------|--------|-----------|
| 50        | Soft nudge injected into stream | Low-cost course correction |
| 500       | Parallel self-check (diagnostic) | First probe, no action taken |
| 1000      | "STOP! Restart with fresh context or you will be terminated." | Warning, NOT a kill |
| 1500      | Self-confirmation gate: model reviews progress, answers true/false. TRUE→kill, FALSE→continue | Confirmation gate prevents false kills |
| 3000      | Unconditional kill | Final safety net |

Self-check failure or timeout → continue, not kill. The cost of a false
kill (killing a valid process making real progress) is higher than the
cost of letting a stuck process run until the next threshold.

### Component contracts

**Model** — `query(messages) -> response`
Wraps any LLM endpoint. Handles streaming, token accounting, thinking-level
control, stop sequences. Subclasses: `OpenAIModel` (llama.cpp/vLLM/Ollama),
`LitellmModel` (hosted APIs), `CliModel` (claude-cli, gemini, copilot).

**Environment** — `execute(action) -> observation`
Runs the agent's commands in a sandbox. Subclasses: `LocalEnv`,
`DockerEnv`, `KubectlEnv`, `SandboxedEnv` (bubblewrap).

**Observer** — `check(messages, response) -> Optional[Violation]`
Inspects the conversation for degenerate patterns. Returns a `Violation`
enum: `DOOM_LOOP`, `TEXT_REPETITION`, `FORMAT_ERROR`, `CONTEXT_PRESSURE`.
The agent decides what to do based on the violation type.

**Journal** — `record(command, exit_code, output)`, `summary() -> str`
Disk-persisted record of what the agent tried, what worked, and what
didn't. Survives context resets. Fed to the model at the start of each
fresh-context iteration.

---

## Feature Catalog

### From mini-swe-agent (the base)

| Feature | Description |
|---|---|
| Linear history | Every step appends to `messages`. The trajectory IS the message list. No hidden state, trivially debuggable. |
| Stateless execution | `subprocess.run` per action, not a persistent shell. No stateful session can accumulate and go wrong. |
| `max_consecutive_format_errors` | Exit after N format errors in a row. Catches degenerate output structurally — garbage doesn't parse. |
| `step_limit` | Hard stop on number of LLM calls. |
| `cost_limit` | Hard stop on accumulated API cost. |
| `wall_time_limit_seconds` | Hard stop on elapsed time. |
| `role: "exit"` protocol | The loop breaks on an exit message. Model can signal completion, or any limit/error forces one. |
| Config via Pydantic | Type-safe config with defaults, validation, and YAML/CLI/env merging. |
| Trajectory serialization | Full message history + metadata saved as JSON for debugging and fine-tuning. |

### From Ralph Wiggum (anti-loop methodology)

| Feature | Description |
|---|---|
| Fresh-context reset | On degenerate loop detection, kill the context and restart with a fresh message list + journal summary. The model can't loop on identical context if the context is always fresh. |
| Disk-persisted state | The `Journal` lives on disk, not in the model's context. State survives resets without context bloat. |
| Completion signal | Structured `<done>` tag (not free-form SUBMIT) that the harness greps for. Reliable, unambiguous. |
| `max_retries` tracking | After N attempts on a task, flag as "stuck" and abandon. Task-level repetition detection. |
| Consecutive failure detection | 3 iterations without completion → warning to simplify the task or fix blockers. |

### From ZeroStack (loop detection)

| Feature | Description |
|---|---|
| Doom-loop detection | 3 identical tool calls in a row → `DOOM_LOOP` violation. Catches the 7,303-repetition case after 3, not 7,303. |
| Mid-turn compaction | Compact context at 80% pressure during a turn, not just between turns. Prevents context overflow mid-generation. |
| Sandbox mode | Bubblewrap/network-isolated execution. The agent's commands cannot reach the host. |

### From FestBench analysis (hard-won fixes)

| Feature | Description |
|---|---|
| Stop sequences | `stop=["User:", "\nUser:", "User:\n"]` on every API call. Prevents conversation simulation at the API level. |
| In-stream repetition detection | Rolling 50-token window; if the same sequence appears >5 times, abort the HTTP connection mid-stream. Catches degenerate output within a single API call. |
| Proper messages array | Send `{role, content}` messages directly — never flatten to plain text with `User:`/`Assistant:` markers. The model never sees the pattern it would copy. |
| Thinking level control | `chat_template_kwargs` with `enable_thinking` and `reasoning_effort`. The core measurement dimension for reasoning models. |
| Per-call maxtok by level | off:4000, low:8000, medium:16000, xhigh:12000. Prevents one degenerate turn from eating the entire budget. |
| Response truncation | Stored assistant responses capped at 2000 chars after action extraction. Prevents context poisoning from degenerate output. |
| Temperature flag | Default 0 for reproducibility; `--temperature` flag for testing loop-breaking with sampling randomness. |

### New: Trial Journal (stored memory)

The Journal is `zubble`'s distinctive feature — a persistent, structured
record of what the agent tried, what worked, and what didn't. This
prevents the agent from repeating failed approaches after a context
reset, and builds an evolving knowledge base across runs.

**Per-entry fields:**
- `command` — the shell command attempted
- `exit_code` — 0 = worked, non-zero = failed
- `result_summary` — first 200 chars of output (enough to recognize, not enough to bloat)
- `category` — `diagnostic` | `fix_attempt` | `verification` (auto-classified)
- `model_note` — optional note the model can add about why it failed
- `timestamp` — when the command was run

**Storage:** `{work_dir}/.zubble/journal.json` — one file per task, cleared on new task.

**Fed to model on context reset:**
```
[Previous attempts in this task — try a different approach]

DIAGNOSTIC:
  ✗ kubectl get pods -A → connection refused
  ✗ kubectl get nodes → connection refused
  ✓ kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes → Ready

FIX_ATTEMPT:
  ✗ kubectl patch deployment ... → forbidden (insufficient RBAC)
  ✓ kubectl edit deployment ... → updated

VERIFICATION:
  ✓ kubectl rollout status deployment/xavier-ocr → deployment successfully rolled out

[Note: direct API access requires the admin kubeconfig at /etc/kubernetes/admin.conf]
```

**Cross-run memory (optional):** If `--memory-dir` is set, the journal
is also saved to a shared directory keyed by task type. Future runs of
similar tasks can load prior journals as a starting knowledge base. This
creates an evolving "what worked for this type of problem" library.

---

## Module Layout

```
zubble/
├── LICENSE                      # MIT
├── README.md
├── DESIGN.md                    # This document
├── pyproject.toml               # pip install zubble-agent
├── zubble/
│   ├── __init__.py
│   ├── agent.py                 # ~200 lines: the core loop
│   ├── config.py                # Pydantic config models
│   ├── exceptions.py            # FormatError, LimitsExceeded, etc.
│   │
│   ├── models/                  # Pluggable LLM backends
│   │   ├── __init__.py          # Model base class
│   │   ├── openai_model.py      # OpenAI-compatible (llama.cpp, vLLM, Ollama)
│   │   ├── litellm_model.py     # Hosted APIs (Anthropic, OpenAI, etc.)
│   │   └── cli_model.py         # Local CLIs (claude, gemini, copilot)
│   │
│   ├── environments/            # Pluggable execution sandboxes
│   │   ├── __init__.py          # Environment base class
│   │   ├── local_env.py         # subprocess.run on local machine
│   │   ├── docker_env.py        # docker exec
│   │   ├── kubectl_env.py       # kubectl exec into a pod
│   │   └── sandboxed_env.py     # bubblewrap isolation
│   │
│   ├── observers/               # Pluggable loop detectors
│   │   ├── __init__.py          # Observer base class, Violation enum
│   │   ├── doom_loop.py         # 3 identical commands = trigger
│   │   ├── text_repetition.py   # In-stream and cross-turn text repetition
│   │   └── format_errors.py     # max_consecutive_format_errors
│   │
│   ├── journal.py               # TrialJournal: disk-persisted memory
│   │
│   ├── protocols/               # Task-specific message templates
│   │   ├── __init__.py
│   │   ├── repair.py            # Infrastructure repair (FestBench-style)
│   │   ├── swe.py               # SWE-bench-style coding tasks
│   │   └── custom.py            # User-defined task protocol
│   │
│   └── run.py                   # CLI entry point: `zubble run --task ...`
│
└── tests/
    ├── test_agent.py
    ├── test_observers.py
    ├── test_journal.py
    └── test_models.py
```

---

## Agent Control Flow

```python
class Agent:
    def run(self, task: str) -> RunResult:
        self.messages = self.protocol.initial_messages(task, self.journal)
        self.journal.start(task)

        while True:
            # 1. Check hard limits
            self._check_limits()           # step, cost, wall_time

            # 2. Query the model (with stop sequences, thinking control)
            response = self.model.query(
                self.messages,
                stop=self.protocol.stop_sequences,
                max_tokens=self.protocol.maxtok_for_level(self.level),
                chat_template_kwargs=self.level_kwargs,
                temperature=self.temperature,
            )
            self.messages.append(response.message)
            self.cost += response.cost
            self.n_calls += 1

            # 3. Check for completion signal
            if self.protocol.is_complete(response.text):
                return RunResult(status="completed", ...)

            # 4. Extract and execute actions
            actions = self.protocol.parse_actions(response.text)
            if not actions:
                self.n_consecutive_format_errors += 1
                if self.n_consecutive_format_errors >= self.config.max_consecutive_format_errors:
                    raise FormatError("too many format errors in a row")
                continue
            self.n_consecutive_format_errors = 0

            for action in actions:
                obs = self.env.execute(action)
                self.journal.record(action, obs)
                self.messages.append(self.protocol.format_observation(obs))

            # 5. Run observers (loop detection)
            violation = self.observer.check(self.messages, response)
            if violation:
                if violation.type == Violation.DOOM_LOOP:
                    self._handle_doom_loop(violation)
                elif violation.type == Violation.TEXT_REPETITION:
                    self._handle_context_reset(violation)
                elif violation.type == Violation.FORMAT_ERROR:
                    self._handle_format_error(violation)

            # 6. Truncate stored response to prevent context poisoning
            self._truncate_last_response(max_chars=2000)

    def _handle_context_reset(self, violation):
        """Ralph Wiggum-style: fresh context, preserved journal."""
        self.context_resets += 1
        if self.context_resets > self.config.max_context_resets:
            raise LimitsExceeded("exhausted context resets")

        journal_summary = self.journal.summary()
        self.messages = self.protocol.reset_messages(
            task=self.task,
            journal=journal_summary,
            reason=str(violation),
        )
        # Reset per-loop tracking but keep the journal
        self.n_consecutive_format_errors = 0
```

---

## CLI Usage (standalone, no FestBench)

```bash
# Install
pip install zubble-agent

# Run against any OpenAI-compatible endpoint
zubble run \
  --endpoint http://192.168.5.117:8000 \
  --model nemotron-3.5-bolt-74b \
  --task "fix the kubelet config on node-1" \
  --env docker \
  --env-container bastion-03 \
  --level xhigh \
  --minutes 15 \
  --stop-sequences "User:" "\nUser:" \
  --temperature 0

# Run against a hosted API
zubble run \
  --model anthropic/claude-sonnet-4 \
  --task "refactor the auth module" \
  --env local \
  --protocol swe

# Run with trial journal + cross-run memory
zubble run \
  --task "debug the cert_expiry_triage scenario" \
  --env kubectl \
  --memory-dir ~/.zubble/memory \
  --journal

# Run as a Ralph Wiggum loop (fresh context each iteration)
zubble loop \
  --task spec.md \
  --max-iterations 20 \
  --fresh-context \
  --completion-signal "<done>"
```

---

## Integration with FestBench

`zubble` is designed to run independently, but FestBench can use it as a
drop-in replacement for `scenariobench.py`'s `make_backend`:

```python
# In scenariobench.py (future integration)
from zubble import Agent, OpenAIModel, DockerEnv, RepairProtocol, Journal

def make_backend(via, model, base, container=None):
    if via == "zubble":
        env = DockerEnv(container=container)
        mdl = OpenAIModel(endpoint=base, model=model)
        protocol = RepairProtocol(report_path="/findings/report.md")
        journal = Journal(work_dir=f"/tmp/festbench_{container}")

        def gen(messages, timeout, kwargs=None, maxtok=4000, stop=None):
            agent = Agent(mdl, env, protocol, journal, level_kwargs=kwargs)
            result = agent.run_task(messages, timeout=timeout, maxtok=maxtok)
            return result.text, None, result.perf
        return gen
```

FestBench's scenario orchestration (kind clusters, oracles, CWD tracking)
stays in `scenariobench.py`. `zubble` handles only the agent loop — the part
that was broken.

---

## Comparison

The four harnesses below cover the corners of the industry design space
`zubble` distances itself from. **OpenHands** is the heavyweight multi-agent
platform — web UI, Agent-Client Protocol, Docker sandboxes, automations,
and a multi-repo TypeScript/Python codebase. **Aider** is the terminal
pair-programmer: git-integrated, litellm-backed, with an architect/editor
split for reasoning-then-edit workflows. **opencode** is the
permission-driven CLI agent — MCP support, subagents, and a built-in
`doom_loop` permission that triggers when the same tool call repeats three
times. **pi.dev** is a minimal, extensible TypeScript harness with skills,
extensions, packages, MCP and ACP support; it is also the harness FestBench
currently runs via `pi-harness:0.85.1`, so the `pi.dev` column shows
exactly what `zubble` adds beyond the status quo.

| Feature | mini-swe-agent | Ralph Wiggum | ZeroStack | OpenHands | Aider | opencode | pi.dev | **zubble** |
|---|---|---|---|---|---|---|---|---|
| Core LoC | ~200 | ~400 (bash) | ~30K (Rust) | ~100K+ (TS/Py) | ~50K (Py) | ~100K (TS) | ~50K (TS) | **~200 (Py)** |
| License | MIT | MIT | GPL-3.0 | MIT | Apache-2.0 | MIT | MIT | **MIT** |
| Stop sequences | partial | N/A | no | no | no | no | no | **yes** |
| In-stream repetition detection | no | no | no | no | no | no | no | **yes** |
| Proper messages array | yes | N/A | yes | yes | yes | yes | yes | **yes** |
| Doom-loop detection (3x) | no | no | yes | partial | partial | yes (built-in `doom_loop` permission) | no | **yes** |
| Fresh-context reset | no | yes (core) | yes | no | no | partial (compaction) | partial (auto-compaction + branch summary) | **yes** |
| Trial journal | no | no | no | no | no (git commits) | no (snapshots) | no (sessions JSONL is telemetry, not fed back) | **yes** |
| Cross-run memory | no | no | no | no | no | yes (memories + skills) | yes (skills, extensions, packages) | **yes** (optional) |
| Thinking level control | partial | N/A | no | yes | yes | no | yes (per-model overrides) | **yes** |
| Per-call maxtok by level | no | N/A | no | no | no | no | no | **yes** |
| Wall-time / step / cost limits | yes | yes | yes | yes | partial | no | partial (overflow recovery, context budget) | **yes** |
| Stateless execution | yes | N/A | yes | no | no | no | no | **yes** |
| Sandbox isolation | partial | N/A | yes (bubblewrap) | yes (Docker) | no (host fs) | partial (permissions) | no (no built-in sandbox) | **yes** |
| Pluggable environments | yes | no | no | yes (backends) | no | no | no | **yes** |
| Pluggable models | yes (litellm) | no (CLIs only) | yes | yes | yes (litellm) | yes (providers) | yes (providers, llama.cpp, custom) | **yes** |
| FestBench integration | no | no | no | no | no | no | yes (`pi-harness:0.85.1`, `--via pi`) | **yes** (optional) |
| Standalone operation | yes | yes | yes | yes | yes | yes | yes | **yes** |
| GUI / TUI | CLI | CLI | CLI | Web UI | TUI | TUI + desktop | TUI (+ print/JSON/RPC/SDK) | CLI |
| MCP / ACP support | no | no | no | ACP | no | MCP | both | no |
| Multi-agent orchestration | no | no | no | yes (automations + agent server) | partial (architect/editor split) | yes (subagents, `subagent_depth`) | no (explicit; skips sub-agents and plan mode) | no |

---

## Roadmap

### Phase 0: Skeleton (this document)
- [x] `zubble/` directory created in llm_benchmark repo
- [x] LICENSE (MIT)
- [x] DESIGN.md (this file)

### Phase 1: Structured protocol + plan-state ledger
- [x] `protocol.py` — JSON schema, system prompt template, parse-retry wrapper
- [x] `ledger.py` — plan state machine, attempted actions, abandoned approaches
      with causal failure reasons, divergence detection, summary injection
- [x] `stream_chat` in benchlib.py — accepts optional `system_prompt` and
      `response_format` (defaults None; existing callers unaffected)
- [x] `scenariobench.py` integration — `--structured` flag enables JSON protocol
      + ledger injection; existing text protocol preserved as default
- [ ] Tests: re-run 5 xhigh DepthSweep scenarios, compare against 7303-rep baseline

### Phase 2: In-stream repetition detection
- [ ] `repetition.py` — rolling 10-gram CRC32 fingerprint, last-200-token window
- [ ] Graduated thresholds:
      50 reps  → soft nudge injected into stream
      500 reps → parallel self-check (diagnostic, no action)
      1000 reps → "STOP! Restart with fresh context or you will be terminated."
                  (warning only, does NOT kill)
      1500 reps → self-confirmation gate: model reviews progress, answers
                  true/false. TRUE → kill. FALSE → continue.
      3000 reps → unconditional kill (final safety net)
- [ ] Self-check failure/timeout → continue, not kill
- [ ] Tests for repetition detection at each threshold

### Phase 3: Ledger hierarchical summarization
- [ ] Category buckets (network/compute/storage/auth/rbac/cert)
- [ ] Collapse oldest entries into summary when `attempted` > 20
- [ ] Context reset with ledger summary on 1500-rep kill

### Phase 4: CLI + Polish
- [ ] `run.py` — CLI entry point
- [ ] `pyproject.toml` — pip installable
- [ ] `README.md` with quickstart
- [ ] Integration test: run one FestBench scenario end-to-end

### Phase 5 (deferred): Web search
- [ ] Per-scenario opt-in flag in ScenarioBench YAML
- [ ] DuckDuckGo/Brave/SearXNG fetch + model-extracted relevant steps
- [ ] Measure search+apply vs pure-reasoning separately

---

## Attribution

`zubble` draws heavily from:
- [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) by Princeton/Stanford (MIT)
- [Ralph Wiggum](https://github.com/fstandhartinger/ralph-wiggum) by fstandhartinger (MIT), based on Geoffrey Huntley's methodology
- [ZeroStack](https://github.com/gi-dellav/zerostack) by gi-dellav (GPL-3.0) — doom-loop and compaction patterns
- FestBench's `scenariobench.py` — the hard-won lessons from degenerate loops