zubble/DESIGN.md

zubble — A Minimal Agentic Harness

A hackable, extensible agent loop that combines the simplicity of mini-swe-agent with the anti-loop discipline of Ralph Wiggum, the doom-loop detection of ZeroStack, and hard-won lessons from FestBench's ScenarioBench harness.

---

Design Principles

1. Readable. The core agent loop is ~200 lines of Python. A new contributor should understand the entire control flow in one sitting. No magic, no metaprogramming, no framework tax.

2. Extensible. Three pluggable interfaces — Model, Environment, Observer — cover every extension point. New providers, sandboxes, or grading hooks are subclasses, not config gymnastics.

3. Sharable. MIT licensed. No FestBench dependency. Runs standalone against any OpenAI-compatible endpoint, hosted API, or local CLI. Pip-installable. Single-file deployable.

4. Pythonic. Type-hinted, dataclass configs, stdlib-first. Pydantic for config validation (same as mini-swe-agent). No Rust, no Go, no Node — the language the research community already speaks.

5. Honest. Every failure mode is recorded, not hidden. Degenerate loops are detected and logged, not silently retried. Token costs are tracked per call, not estimated after the fact.

---

Why "zubble"?

Named after the world's best dog. A good agentic harness is Man's best friend — loyal, dependable, always ready to help you fix what's broken.

---

Origin: What We Learned from FestBench

FestBench's scenariobench.py gave a model a broken Kubernetes cluster, a wall-clock budget, and a free-form text protocol. At xhigh thinking levels, the model generated 79K characters of degenerate conversation simulation — emitting both User: and Assistant: turns in a single response, 7,303 repetitions of the word "truncated" before timing out.

Five root causes were identified:

1. No stop sequences — the API call had no stop=["User:"], so generation continued through simulated conversation turns.
2. Flat-text rendering — the harness flattened the message history into a single string with User:/Assistant: markers, then sent it as one user message. The model saw the pattern and copied it.
3. No in-stream repetition detection — the livelock detector only checked extracted commands between turns, not repeated text within a single generation.
4. Unbounded maxtok — xhigh's 32K token budget at 41.5 tok/s = 770s, longer than the 480s timeout. One degenerate turn ate the entire scenario budget.
5. No context reset — when the model got stuck, the accumulated context (including the 79K-char degenerate response) poisoned every subsequent turn.

zubble is designed to make all five of these failure modes structurally impossible.

---

Architecture

stream_chat()              ← shared HTTP client (unchanged, no ledger awareness)
    ↑
    ├── softbench.py           → calls stream_chat once per task, scores output
    ├── infrabench.py          → calls stream_chat once per task, validates output
    ├── prosebench.py          → calls stream_chat once per task, scores output
    ├── psychbench.py          → calls stream_chat once per task, scores output
    └── scenariobench.py       → calls agent_loop() which calls stream_chat N times
                                    ↑
                              agent_loop (zubble)
                                  ├── protocol.py     ← JSON schema, system prompt, parse-retry
                                  ├── ledger.py       ← plan state machine (short-term memory)
                                  └── repetition.py   ← in-stream repetition detection

The plan-state ledger is the model's short-term memory. It tracks not just what was attempted, but the full causal chain: what the model planned to do, which steps are done vs pending, where execution diverged from the plan, and why previous approaches were abandoned.

Protocol

The model communicates in structured JSON, not free-form text:

{
  "action": "plan" | "run" | "report" | "submit" | "ask",
  "command": "<shell command, for action=run>",
  "plan_step_id": <int, references the plan>,
  "reasoning": "<brief explanation>",
  "steps": ["<step 1>", "<step 2>", ...],  // for action=plan only
  "report": "<short submission summary>",   // for action=report/submit only
  "question": "<clarification>",            // for action=ask only
  "revised_plan": ["<new steps>"]           // when abandoning current plan
}

Turn 1: model emits action: "plan" with an ordered list of steps. Turns 2..N: model emits action: "run" with plan_step_id referencing which step this action fulfills. The harness checks each action against the plan state machine. If the action doesn't match the next pending step, a divergence warning is injected before the command executes.

submit is the only verification path. A rejected submission tells the model to keep working and submit again; the harness owns termination through the deadline, turn cap, or doom-loop safeguards. The full post-mortem belongs in the scenario report file, while the JSON report field is a short, single-line summary.

JSON enforcement combines endpoint response_format: json_schema, strict single-object parsing, and one targeted repair generation on malformed output. After a failed repair, the scenario exits as parse_error with the parser reason rather than spending its remaining budget in an invalid-JSON loop.

Plan-state ledger

PLAN (captured at turn 1):
  Step 1: Diagnose bad cert           → DONE (found expired cert on node-b)
  Step 2: Create new cert manifest    → DONE (cert-fix.yaml created)
  Step 3: Apply manifest              → PENDING
  Step 4: Verify cert renewed         → PENDING

ABANDONED APPROACHES (with causal failure reasons):
  - "restart kubelet" → failed because kubelet was already running;
    cert expiry was root cause, not service state

CURRENT STATE:
  Next expected step: Step 3 (apply manifest)
  Last action: kubectl create -f cert-fix.yaml (step 2, succeeded)
  Divergence flag: NONE

Injected into every prompt as "## SESSION MEMORY" so the model sees its plan, progress, and abandoned approaches before choosing its next action.

Repetition detection (Phase 2)

Rolling 10-gram CRC32 fingerprint over the last 200 generated tokens. Graduated thresholds prevent false-positive kills of valid long-running processes:

Threshold	Action	Rationale
50	Soft nudge injected into stream	Low-cost course correction
500	Parallel self-check (diagnostic)	First probe, no action taken
1000	"STOP! Restart with fresh context or you will be terminated."	Warning, NOT a kill
1500	Self-confirmation gate: model reviews progress, answers true/false. TRUE→kill, FALSE→continue	Confirmation gate prevents false kills
3000	Unconditional kill	Final safety net

Self-check failure or timeout → continue, not kill. The cost of a false kill (killing a valid process making real progress) is higher than the cost of letting a stuck process run until the next threshold.

Component contracts

Model — query(messages) -> response Wraps any LLM endpoint. Handles streaming, token accounting, thinking-level control, stop sequences. Subclasses: OpenAIModel (llama.cpp/vLLM/Ollama), LitellmModel (hosted APIs), CliModel (claude-cli, gemini, copilot).

Environment — execute(action) -> observation Runs the agent's commands in a sandbox. Subclasses: LocalEnv, DockerEnv, KubectlEnv, SandboxedEnv (bubblewrap).

Observer — check(messages, response) -> Optional[Violation] Inspects the conversation for degenerate patterns. Returns a Violation enum: DOOM_LOOP, TEXT_REPETITION, FORMAT_ERROR, CONTEXT_PRESSURE. The agent decides what to do based on the violation type.

Journal — record(command, exit_code, output), summary() -> str Disk-persisted record of what the agent tried, what worked, and what didn't. Survives context resets. Fed to the model at the start of each fresh-context iteration.

---

Feature Catalog

From mini-swe-agent (the base)

Feature	Description
Linear history	Every step appends to messages. The trajectory IS the message list. No hidden state, trivially debuggable.
Stateless execution	subprocess.run per action, not a persistent shell. No stateful session can accumulate and go wrong.
max_consecutive_format_errors	Exit after N format errors in a row. Catches degenerate output structurally — garbage doesn't parse.
step_limit	Hard stop on number of LLM calls.
cost_limit	Hard stop on accumulated API cost.
wall_time_limit_seconds	Hard stop on elapsed time.
role: "exit" protocol	The loop breaks on an exit message. Model can signal completion, or any limit/error forces one.
Config via Pydantic	Type-safe config with defaults, validation, and YAML/CLI/env merging.
Trajectory serialization	Full message history + metadata saved as JSON for debugging and fine-tuning.

From Ralph Wiggum (anti-loop methodology)

Feature	Description
Fresh-context reset	On degenerate loop detection, kill the context and restart with a fresh message list + journal summary. The model can't loop on identical context if the context is always fresh.
Disk-persisted state	The Journal lives on disk, not in the model's context. State survives resets without context bloat.
Completion signal	Structured <done> tag (not free-form SUBMIT) that the harness greps for. Reliable, unambiguous.
max_retries tracking	After N attempts on a task, flag as "stuck" and abandon. Task-level repetition detection.
Consecutive failure detection	3 iterations without completion → warning to simplify the task or fix blockers.

From ZeroStack (loop detection)

Feature	Description
Doom-loop detection	3 identical tool calls in a row → DOOM_LOOP violation. Catches the 7,303-repetition case after 3, not 7,303.
Mid-turn compaction	Compact context at 80% pressure during a turn, not just between turns. Prevents context overflow mid-generation.
Sandbox mode	Bubblewrap/network-isolated execution. The agent's commands cannot reach the host.

From FestBench analysis (hard-won fixes)

Feature	Description
Stop sequences	stop=["User:", "\nUser:", "User:\n"] on every API call. Prevents conversation simulation at the API level.
In-stream repetition detection	Rolling 50-token window; if the same sequence appears >5 times, abort the HTTP connection mid-stream. Catches degenerate output within a single API call.
Proper messages array	Send {role, content} messages directly — never flatten to plain text with User:/Assistant: markers. The model never sees the pattern it would copy.
Thinking level control	chat_template_kwargs with enable_thinking and reasoning_effort. The core measurement dimension for reasoning models.
Per-call maxtok by level	off:4000, low:8000, medium:16000, xhigh:12000. Prevents one degenerate turn from eating the entire budget.
Response truncation	Stored assistant responses capped at 2000 chars after action extraction. Prevents context poisoning from degenerate output.
Temperature flag	Default 0 for reproducibility; --temperature flag for testing loop-breaking with sampling randomness.

New: Trial Journal (stored memory)

The Journal is zubble's distinctive feature — a persistent, structured record of what the agent tried, what worked, and what didn't. This prevents the agent from repeating failed approaches after a context reset, and builds an evolving knowledge base across runs.

Per-entry fields:

• command — the shell command attempted
• exit_code — 0 = worked, non-zero = failed
• result_summary — first 200 chars of output (enough to recognize, not enough to bloat)
• category — diagnostic | fix_attempt | verification (auto-classified)
• model_note — optional note the model can add about why it failed
• timestamp — when the command was run

Storage: {work_dir}/.zubble/journal.json — one file per task, cleared on new task.

Fed to model on context reset:

[Previous attempts in this task — try a different approach]

DIAGNOSTIC:
  ✗ kubectl get pods -A → connection refused
  ✗ kubectl get nodes → connection refused
  ✓ kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes → Ready

FIX_ATTEMPT:
  ✗ kubectl patch deployment ... → forbidden (insufficient RBAC)
  ✓ kubectl edit deployment ... → updated

VERIFICATION:
  ✓ kubectl rollout status deployment/xavier-ocr → deployment successfully rolled out

[Note: direct API access requires the admin kubeconfig at /etc/kubernetes/admin.conf]

Cross-run memory (optional): If --memory-dir is set, the journal is also saved to a shared directory keyed by task type. Future runs of similar tasks can load prior journals as a starting knowledge base. This creates an evolving "what worked for this type of problem" library.

---

Module Layout

zubble/
├── LICENSE                      # MIT
├── README.md
├── DESIGN.md                    # This document
├── pyproject.toml               # pip install zubble-agent
├── zubble/
│   ├── __init__.py
│   ├── agent.py                 # ~200 lines: the core loop
│   ├── config.py                # Pydantic config models
│   ├── exceptions.py            # FormatError, LimitsExceeded, etc.
│   │
│   ├── models/                  # Pluggable LLM backends
│   │   ├── __init__.py          # Model base class
│   │   ├── openai_model.py      # OpenAI-compatible (llama.cpp, vLLM, Ollama)
│   │   ├── litellm_model.py     # Hosted APIs (Anthropic, OpenAI, etc.)
│   │   └── cli_model.py         # Local CLIs (claude, gemini, copilot)
│   │
│   ├── environments/            # Pluggable execution sandboxes
│   │   ├── __init__.py          # Environment base class
│   │   ├── local_env.py         # subprocess.run on local machine
│   │   ├── docker_env.py        # docker exec
│   │   ├── kubectl_env.py       # kubectl exec into a pod
│   │   └── sandboxed_env.py     # bubblewrap isolation
│   │
│   ├── observers/               # Pluggable loop detectors
│   │   ├── __init__.py          # Observer base class, Violation enum
│   │   ├── doom_loop.py         # 3 identical commands = trigger
│   │   ├── text_repetition.py   # In-stream and cross-turn text repetition
│   │   └── format_errors.py     # max_consecutive_format_errors
│   │
│   ├── journal.py               # TrialJournal: disk-persisted memory
│   │
│   ├── protocols/               # Task-specific message templates
│   │   ├── __init__.py
│   │   ├── repair.py            # Infrastructure repair (FestBench-style)
│   │   ├── swe.py               # SWE-bench-style coding tasks
│   │   └── custom.py            # User-defined task protocol
│   │
│   └── run.py                   # CLI entry point: `zubble run --task ...`
│
└── tests/
    ├── test_agent.py
    ├── test_observers.py
    ├── test_journal.py
    └── test_models.py

---

Agent Control Flow

class Agent:
    def run(self, task: str) -> RunResult:
        self.messages = self.protocol.initial_messages(task, self.journal)
        self.journal.start(task)

        while True:
            # 1. Check hard limits
            self._check_limits()           # step, cost, wall_time

            # 2. Query the model (with stop sequences, thinking control)
            response = self.model.query(
                self.messages,
                stop=self.protocol.stop_sequences,
                max_tokens=self.protocol.maxtok_for_level(self.level),
                chat_template_kwargs=self.level_kwargs,
                temperature=self.temperature,
            )
            self.messages.append(response.message)
            self.cost += response.cost
            self.n_calls += 1

            # 3. Check for completion signal
            if self.protocol.is_complete(response.text):
                return RunResult(status="completed", ...)

            # 4. Extract and execute actions
            actions = self.protocol.parse_actions(response.text)
            if not actions:
                self.n_consecutive_format_errors += 1
                if self.n_consecutive_format_errors >= self.config.max_consecutive_format_errors:
                    raise FormatError("too many format errors in a row")
                continue
            self.n_consecutive_format_errors = 0

            for action in actions:
                obs = self.env.execute(action)
                self.journal.record(action, obs)
                self.messages.append(self.protocol.format_observation(obs))

            # 5. Run observers (loop detection)
            violation = self.observer.check(self.messages, response)
            if violation:
                if violation.type == Violation.DOOM_LOOP:
                    self._handle_doom_loop(violation)
                elif violation.type == Violation.TEXT_REPETITION:
                    self._handle_context_reset(violation)
                elif violation.type == Violation.FORMAT_ERROR:
                    self._handle_format_error(violation)

            # 6. Truncate stored response to prevent context poisoning
            self._truncate_last_response(max_chars=2000)

    def _handle_context_reset(self, violation):
        """Ralph Wiggum-style: fresh context, preserved journal."""
        self.context_resets += 1
        if self.context_resets > self.config.max_context_resets:
            raise LimitsExceeded("exhausted context resets")

        journal_summary = self.journal.summary()
        self.messages = self.protocol.reset_messages(
            task=self.task,
            journal=journal_summary,
            reason=str(violation),
        )
        # Reset per-loop tracking but keep the journal
        self.n_consecutive_format_errors = 0

---

CLI Usage (standalone, no FestBench)

# Install
pip install zubble-agent

# Run against any OpenAI-compatible endpoint
zubble run \
  --endpoint http://192.168.5.117:8000 \
  --model nemotron-3.5-bolt-74b \
  --task "fix the kubelet config on node-1" \
  --env docker \
  --env-container bastion-03 \
  --level xhigh \
  --minutes 15 \
  --stop-sequences "User:" "\nUser:" \
  --temperature 0

# Run against a hosted API
zubble run \
  --model anthropic/claude-sonnet-4 \
  --task "refactor the auth module" \
  --env local \
  --protocol swe

# Run with trial journal + cross-run memory
zubble run \
  --task "debug the cert_expiry_triage scenario" \
  --env kubectl \
  --memory-dir ~/.zubble/memory \
  --journal

# Run as a Ralph Wiggum loop (fresh context each iteration)
zubble loop \
  --task spec.md \
  --max-iterations 20 \
  --fresh-context \
  --completion-signal "<done>"

---

Integration with FestBench

zubble is designed to run independently, but FestBench can use it as a drop-in replacement for scenariobench.py's make_backend:

# In scenariobench.py (future integration)
from zubble import Agent, OpenAIModel, DockerEnv, RepairProtocol, Journal

def make_backend(via, model, base, container=None):
    if via == "zubble":
        env = DockerEnv(container=container)
        mdl = OpenAIModel(endpoint=base, model=model)
        protocol = RepairProtocol(report_path="/findings/report.md")
        journal = Journal(work_dir=f"/tmp/festbench_{container}")

        def gen(messages, timeout, kwargs=None, maxtok=4000, stop=None):
            agent = Agent(mdl, env, protocol, journal, level_kwargs=kwargs)
            result = agent.run_task(messages, timeout=timeout, maxtok=maxtok)
            return result.text, None, result.perf
        return gen

FestBench's scenario orchestration (kind clusters, oracles, CWD tracking) stays in scenariobench.py. zubble handles only the agent loop — the part that was broken.

---

Comparison

The four harnesses below cover the corners of the industry design space zubble distances itself from. OpenHands is the heavyweight multi-agent platform — web UI, Agent-Client Protocol, Docker sandboxes, automations, and a multi-repo TypeScript/Python codebase. Aider is the terminal pair-programmer: git-integrated, litellm-backed, with an architect/editor split for reasoning-then-edit workflows. opencode is the permission-driven CLI agent — MCP support, subagents, and a built-in doom_loop permission that triggers when the same tool call repeats three times. pi.dev is a minimal, extensible TypeScript harness with skills, extensions, packages, MCP and ACP support; it is also the harness FestBench currently runs via pi-harness:0.85.1, so the pi.dev column shows exactly what zubble adds beyond the status quo.

Feature	mini-swe-agent	Ralph Wiggum	ZeroStack	OpenHands	Aider	opencode	pi.dev	zubble
Core LoC	~200	~400 (bash)	~30K (Rust)	~100K+ (TS/Py)	~50K (Py)	~100K (TS)	~50K (TS)	~200 (Py)
License	MIT	MIT	GPL-3.0	MIT	Apache-2.0	MIT	MIT	MIT
Stop sequences	partial	N/A	no	no	no	no	no	yes
In-stream repetition detection	no	no	no	no	no	no	no	yes
Proper messages array	yes	N/A	yes	yes	yes	yes	yes	yes
Doom-loop detection (3x)	no	no	yes	partial	partial	yes (built-in doom_loop permission)	no	yes
Fresh-context reset	no	yes (core)	yes	no	no	partial (compaction)	partial (auto-compaction + branch summary)	yes
Trial journal	no	no	no	no	no (git commits)	no (snapshots)	no (sessions JSONL is telemetry, not fed back)	yes
Cross-run memory	no	no	no	no	no	yes (memories + skills)	yes (skills, extensions, packages)	yes (optional)
Thinking level control	partial	N/A	no	yes	yes	no	yes (per-model overrides)	yes
Per-call maxtok by level	no	N/A	no	no	no	no	no	yes
Wall-time / step / cost limits	yes	yes	yes	yes	partial	no	partial (overflow recovery, context budget)	yes
Stateless execution	yes	N/A	yes	no	no	no	no	yes
Sandbox isolation	partial	N/A	yes (bubblewrap)	yes (Docker)	no (host fs)	partial (permissions)	no (no built-in sandbox)	yes
Pluggable environments	yes	no	no	yes (backends)	no	no	no	yes
Pluggable models	yes (litellm)	no (CLIs only)	yes	yes	yes (litellm)	yes (providers)	yes (providers, llama.cpp, custom)	yes
FestBench integration	no	no	no	no	no	no	yes (pi-harness:0.85.1, --via pi)	yes (optional)
Standalone operation	yes	yes	yes	yes	yes	yes	yes	yes
GUI / TUI	CLI	CLI	CLI	Web UI	TUI	TUI + desktop	TUI (+ print/JSON/RPC/SDK)	CLI
MCP / ACP support	no	no	no	ACP	no	MCP	both	no
Multi-agent orchestration	no	no	no	yes (automations + agent server)	partial (architect/editor split)	yes (subagents, subagent_depth)	no (explicit; skips sub-agents and plan mode)	no

---

Roadmap

Phase 0: Skeleton (this document)

• zubble/ directory created in llm_benchmark repo
• LICENSE (MIT)
• DESIGN.md (this file)

Phase 1: Structured protocol + plan-state ledger

• protocol.py — JSON schema, system prompt template, parse-retry wrapper
• ledger.py — plan state machine, attempted actions, abandoned approaches with causal failure reasons, divergence detection, summary injection
• stream_chat in benchlib.py — accepts optional system_prompt and response_format (defaults None; existing callers unaffected)
• scenariobench.py integration — --structured flag enables JSON protocol + ledger injection; existing text protocol preserved as default
• Tests: re-run 5 xhigh DepthSweep scenarios, compare against 7303-rep baseline

Phase 2: In-stream repetition detection

• repetition.py — rolling 10-gram CRC32 fingerprint, last-200-token window
• Graduated thresholds: 50 reps → soft nudge injected into stream 500 reps → parallel self-check (diagnostic, no action) 1000 reps → "STOP! Restart with fresh context or you will be terminated." (warning only, does NOT kill) 1500 reps → self-confirmation gate: model reviews progress, answers true/false. TRUE → kill. FALSE → continue. 3000 reps → unconditional kill (final safety net)
• Self-check failure/timeout → continue, not kill
• Tests for repetition detection at each threshold

Phase 3: Ledger hierarchical summarization

• Category buckets (network/compute/storage/auth/rbac/cert)
• Collapse oldest entries into summary when attempted > 20
• Context reset with ledger summary on 1500-rep kill

Phase 4: CLI + Polish

• run.py — CLI entry point
• pyproject.toml — pip installable
• README.md with quickstart
• Integration test: run one FestBench scenario end-to-end

Phase 5 (deferred): Web search

• Per-scenario opt-in flag in ScenarioBench YAML
• DuckDuckGo/Brave/SearXNG fetch + model-extracted relevant steps
• Measure search+apply vs pure-reasoning separately

---

Attribution

zubble draws heavily from:

• mini-swe-agent by Princeton/Stanford (MIT)
• Ralph Wiggum by fstandhartinger (MIT), based on Geoffrey Huntley's methodology
• ZeroStack by gi-dellav (GPL-3.0) — doom-loop and compaction patterns
• FestBench's scenariobench.py — the hard-won lessons from degenerate loops