# zubble — A Minimal Agentic Harness > A hackable, extensible agent loop that combines the simplicity of > [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) with the > anti-loop discipline of [Ralph Wiggum](https://github.com/fstandhartinger/ralph-wiggum), > the doom-loop detection of [ZeroStack](https://github.com/gi-dellav/zerostack), > and hard-won lessons from FestBench's ScenarioBench harness. --- ## Design Principles 1. **Readable.** The core agent loop is ~200 lines of Python. A new contributor should understand the entire control flow in one sitting. No magic, no metaprogramming, no framework tax. 2. **Extensible.** Three pluggable interfaces — `Model`, `Environment`, `Observer` — cover every extension point. New providers, sandboxes, or grading hooks are subclasses, not config gymnastics. 3. **Sharable.** MIT licensed. No FestBench dependency. Runs standalone against any OpenAI-compatible endpoint, hosted API, or local CLI. Pip-installable. Single-file deployable. 4. **Pythonic.** Type-hinted, dataclass configs, stdlib-first. Pydantic for config validation (same as mini-swe-agent). No Rust, no Go, no Node — the language the research community already speaks. 5. **Honest.** Every failure mode is recorded, not hidden. Degenerate loops are detected and logged, not silently retried. Token costs are tracked per call, not estimated after the fact. --- ## Why "zubble"? Named after the world's best dog. A good agentic harness is Man's best friend — loyal, dependable, always ready to help you fix what's broken. --- ## Origin: What We Learned from FestBench FestBench's `scenariobench.py` gave a model a broken Kubernetes cluster, a wall-clock budget, and a free-form text protocol. At xhigh thinking levels, the model generated 79K characters of degenerate conversation simulation — emitting both `User:` and `Assistant:` turns in a single response, 7,303 repetitions of the word "truncated" before timing out. Five root causes were identified: 1. **No stop sequences** — the API call had no `stop=["User:"]`, so generation continued through simulated conversation turns. 2. **Flat-text rendering** — the harness flattened the message history into a single string with `User:`/`Assistant:` markers, then sent it as one user message. The model saw the pattern and copied it. 3. **No in-stream repetition detection** — the livelock detector only checked extracted commands between turns, not repeated text within a single generation. 4. **Unbounded maxtok** — xhigh's 32K token budget at 41.5 tok/s = 770s, longer than the 480s timeout. One degenerate turn ate the entire scenario budget. 5. **No context reset** — when the model got stuck, the accumulated context (including the 79K-char degenerate response) poisoned every subsequent turn. `zubble` is designed to make all five of these failure modes structurally impossible. --- ## Architecture ``` stream_chat() ← shared HTTP client (unchanged, no ledger awareness) ↑ ├── softbench.py → calls stream_chat once per task, scores output ├── infrabench.py → calls stream_chat once per task, validates output ├── prosebench.py → calls stream_chat once per task, scores output ├── psychbench.py → calls stream_chat once per task, scores output └── scenariobench.py → calls agent_loop() which calls stream_chat N times ↑ agent_loop (zubble) ├── protocol.py ← JSON schema, system prompt, parse-retry ├── ledger.py ← plan state machine (short-term memory) └── repetition.py ← in-stream repetition detection ``` The plan-state ledger is the model's short-term memory. It tracks not just what was attempted, but the full causal chain: what the model planned to do, which steps are done vs pending, where execution diverged from the plan, and why previous approaches were abandoned. ### Protocol The model communicates in structured JSON, not free-form text: ```json { "action": "plan" | "run" | "report" | "submit" | "ask", "command": "", "plan_step_id": , "reasoning": "", "steps": ["", "", ...], // for action=plan only "report": "", // for action=report/submit only "question": "", // for action=ask only "revised_plan": [""] // when abandoning current plan } ``` Turn 1: model emits `action: "plan"` with an ordered list of steps. Turns 2..N: model emits `action: "run"` with `plan_step_id` referencing which step this action fulfills. The harness checks each action against the plan state machine. If the action doesn't match the next pending step, a divergence warning is injected before the command executes. `submit` is the only verification path. A rejected submission tells the model to keep working and submit again; the harness owns termination through the deadline, turn cap, or doom-loop safeguards. The full post-mortem belongs in the scenario report file, while the JSON `report` field is a short, single-line summary. JSON enforcement combines endpoint `response_format: json_schema`, strict single-object parsing, and one targeted repair generation on malformed output. After a failed repair, the scenario exits as `parse_error` with the parser reason rather than spending its remaining budget in an invalid-JSON loop. ### Plan-state ledger ``` PLAN (captured at turn 1): Step 1: Diagnose bad cert → DONE (found expired cert on node-b) Step 2: Create new cert manifest → DONE (cert-fix.yaml created) Step 3: Apply manifest → PENDING Step 4: Verify cert renewed → PENDING ABANDONED APPROACHES (with causal failure reasons): - "restart kubelet" → failed because kubelet was already running; cert expiry was root cause, not service state CURRENT STATE: Next expected step: Step 3 (apply manifest) Last action: kubectl create -f cert-fix.yaml (step 2, succeeded) Divergence flag: NONE ``` Injected into every prompt as "## SESSION MEMORY" so the model sees its plan, progress, and abandoned approaches before choosing its next action. ### Repetition detection (Phase 2) Rolling 10-gram CRC32 fingerprint over the last 200 generated tokens. Graduated thresholds prevent false-positive kills of valid long-running processes: | Threshold | Action | Rationale | |-----------|--------|-----------| | 50 | Soft nudge injected into stream | Low-cost course correction | | 500 | Parallel self-check (diagnostic) | First probe, no action taken | | 1000 | "STOP! Restart with fresh context or you will be terminated." | Warning, NOT a kill | | 1500 | Self-confirmation gate: model reviews progress, answers true/false. TRUE→kill, FALSE→continue | Confirmation gate prevents false kills | | 3000 | Unconditional kill | Final safety net | Self-check failure or timeout → continue, not kill. The cost of a false kill (killing a valid process making real progress) is higher than the cost of letting a stuck process run until the next threshold. ### Component contracts **Model** — `query(messages) -> response` Wraps any LLM endpoint. Handles streaming, token accounting, thinking-level control, stop sequences. Subclasses: `OpenAIModel` (llama.cpp/vLLM/Ollama), `LitellmModel` (hosted APIs), `CliModel` (claude-cli, gemini, copilot). **Environment** — `execute(action) -> observation` Runs the agent's commands in a sandbox. Subclasses: `LocalEnv`, `DockerEnv`, `KubectlEnv`, `SandboxedEnv` (bubblewrap). **Observer** — `check(messages, response) -> Optional[Violation]` Inspects the conversation for degenerate patterns. Returns a `Violation` enum: `DOOM_LOOP`, `TEXT_REPETITION`, `FORMAT_ERROR`, `CONTEXT_PRESSURE`. The agent decides what to do based on the violation type. **Journal** — `record(command, exit_code, output)`, `summary() -> str` Disk-persisted record of what the agent tried, what worked, and what didn't. Survives context resets. Fed to the model at the start of each fresh-context iteration. --- ## Feature Catalog ### From mini-swe-agent (the base) | Feature | Description | |---|---| | Linear history | Every step appends to `messages`. The trajectory IS the message list. No hidden state, trivially debuggable. | | Stateless execution | `subprocess.run` per action, not a persistent shell. No stateful session can accumulate and go wrong. | | `max_consecutive_format_errors` | Exit after N format errors in a row. Catches degenerate output structurally — garbage doesn't parse. | | `step_limit` | Hard stop on number of LLM calls. | | `cost_limit` | Hard stop on accumulated API cost. | | `wall_time_limit_seconds` | Hard stop on elapsed time. | | `role: "exit"` protocol | The loop breaks on an exit message. Model can signal completion, or any limit/error forces one. | | Config via Pydantic | Type-safe config with defaults, validation, and YAML/CLI/env merging. | | Trajectory serialization | Full message history + metadata saved as JSON for debugging and fine-tuning. | ### From Ralph Wiggum (anti-loop methodology) | Feature | Description | |---|---| | Fresh-context reset | On degenerate loop detection, kill the context and restart with a fresh message list + journal summary. The model can't loop on identical context if the context is always fresh. | | Disk-persisted state | The `Journal` lives on disk, not in the model's context. State survives resets without context bloat. | | Completion signal | Structured `` tag (not free-form SUBMIT) that the harness greps for. Reliable, unambiguous. | | `max_retries` tracking | After N attempts on a task, flag as "stuck" and abandon. Task-level repetition detection. | | Consecutive failure detection | 3 iterations without completion → warning to simplify the task or fix blockers. | ### From ZeroStack (loop detection) | Feature | Description | |---|---| | Doom-loop detection | 3 identical tool calls in a row → `DOOM_LOOP` violation. Catches the 7,303-repetition case after 3, not 7,303. | | Mid-turn compaction | Compact context at 80% pressure during a turn, not just between turns. Prevents context overflow mid-generation. | | Sandbox mode | Bubblewrap/network-isolated execution. The agent's commands cannot reach the host. | ### From FestBench analysis (hard-won fixes) | Feature | Description | |---|---| | Stop sequences | `stop=["User:", "\nUser:", "User:\n"]` on every API call. Prevents conversation simulation at the API level. | | In-stream repetition detection | Rolling 50-token window; if the same sequence appears >5 times, abort the HTTP connection mid-stream. Catches degenerate output within a single API call. | | Proper messages array | Send `{role, content}` messages directly — never flatten to plain text with `User:`/`Assistant:` markers. The model never sees the pattern it would copy. | | Thinking level control | `chat_template_kwargs` with `enable_thinking` and `reasoning_effort`. The core measurement dimension for reasoning models. | | Per-call maxtok by level | off:4000, low:8000, medium:16000, xhigh:12000. Prevents one degenerate turn from eating the entire budget. | | Response truncation | Stored assistant responses capped at 2000 chars after action extraction. Prevents context poisoning from degenerate output. | | Temperature flag | Default 0 for reproducibility; `--temperature` flag for testing loop-breaking with sampling randomness. | ### New: Trial Journal (stored memory) The Journal is `zubble`'s distinctive feature — a persistent, structured record of what the agent tried, what worked, and what didn't. This prevents the agent from repeating failed approaches after a context reset, and builds an evolving knowledge base across runs. **Per-entry fields:** - `command` — the shell command attempted - `exit_code` — 0 = worked, non-zero = failed - `result_summary` — first 200 chars of output (enough to recognize, not enough to bloat) - `category` — `diagnostic` | `fix_attempt` | `verification` (auto-classified) - `model_note` — optional note the model can add about why it failed - `timestamp` — when the command was run **Storage:** `{work_dir}/.zubble/journal.json` — one file per task, cleared on new task. **Fed to model on context reset:** ``` [Previous attempts in this task — try a different approach] DIAGNOSTIC: ✗ kubectl get pods -A → connection refused ✗ kubectl get nodes → connection refused ✓ kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes → Ready FIX_ATTEMPT: ✗ kubectl patch deployment ... → forbidden (insufficient RBAC) ✓ kubectl edit deployment ... → updated VERIFICATION: ✓ kubectl rollout status deployment/xavier-ocr → deployment successfully rolled out [Note: direct API access requires the admin kubeconfig at /etc/kubernetes/admin.conf] ``` **Cross-run memory (optional):** If `--memory-dir` is set, the journal is also saved to a shared directory keyed by task type. Future runs of similar tasks can load prior journals as a starting knowledge base. This creates an evolving "what worked for this type of problem" library. --- ## Module Layout ``` zubble/ ├── LICENSE # MIT ├── README.md ├── DESIGN.md # This document ├── pyproject.toml # pip install zubble-agent ├── zubble/ │ ├── __init__.py │ ├── agent.py # ~200 lines: the core loop │ ├── config.py # Pydantic config models │ ├── exceptions.py # FormatError, LimitsExceeded, etc. │ │ │ ├── models/ # Pluggable LLM backends │ │ ├── __init__.py # Model base class │ │ ├── openai_model.py # OpenAI-compatible (llama.cpp, vLLM, Ollama) │ │ ├── litellm_model.py # Hosted APIs (Anthropic, OpenAI, etc.) │ │ └── cli_model.py # Local CLIs (claude, gemini, copilot) │ │ │ ├── environments/ # Pluggable execution sandboxes │ │ ├── __init__.py # Environment base class │ │ ├── local_env.py # subprocess.run on local machine │ │ ├── docker_env.py # docker exec │ │ ├── kubectl_env.py # kubectl exec into a pod │ │ └── sandboxed_env.py # bubblewrap isolation │ │ │ ├── observers/ # Pluggable loop detectors │ │ ├── __init__.py # Observer base class, Violation enum │ │ ├── doom_loop.py # 3 identical commands = trigger │ │ ├── text_repetition.py # In-stream and cross-turn text repetition │ │ └── format_errors.py # max_consecutive_format_errors │ │ │ ├── journal.py # TrialJournal: disk-persisted memory │ │ │ ├── protocols/ # Task-specific message templates │ │ ├── __init__.py │ │ ├── repair.py # Infrastructure repair (FestBench-style) │ │ ├── swe.py # SWE-bench-style coding tasks │ │ └── custom.py # User-defined task protocol │ │ │ └── run.py # CLI entry point: `zubble run --task ...` │ └── tests/ ├── test_agent.py ├── test_observers.py ├── test_journal.py └── test_models.py ``` --- ## Agent Control Flow ```python class Agent: def run(self, task: str) -> RunResult: self.messages = self.protocol.initial_messages(task, self.journal) self.journal.start(task) while True: # 1. Check hard limits self._check_limits() # step, cost, wall_time # 2. Query the model (with stop sequences, thinking control) response = self.model.query( self.messages, stop=self.protocol.stop_sequences, max_tokens=self.protocol.maxtok_for_level(self.level), chat_template_kwargs=self.level_kwargs, temperature=self.temperature, ) self.messages.append(response.message) self.cost += response.cost self.n_calls += 1 # 3. Check for completion signal if self.protocol.is_complete(response.text): return RunResult(status="completed", ...) # 4. Extract and execute actions actions = self.protocol.parse_actions(response.text) if not actions: self.n_consecutive_format_errors += 1 if self.n_consecutive_format_errors >= self.config.max_consecutive_format_errors: raise FormatError("too many format errors in a row") continue self.n_consecutive_format_errors = 0 for action in actions: obs = self.env.execute(action) self.journal.record(action, obs) self.messages.append(self.protocol.format_observation(obs)) # 5. Run observers (loop detection) violation = self.observer.check(self.messages, response) if violation: if violation.type == Violation.DOOM_LOOP: self._handle_doom_loop(violation) elif violation.type == Violation.TEXT_REPETITION: self._handle_context_reset(violation) elif violation.type == Violation.FORMAT_ERROR: self._handle_format_error(violation) # 6. Truncate stored response to prevent context poisoning self._truncate_last_response(max_chars=2000) def _handle_context_reset(self, violation): """Ralph Wiggum-style: fresh context, preserved journal.""" self.context_resets += 1 if self.context_resets > self.config.max_context_resets: raise LimitsExceeded("exhausted context resets") journal_summary = self.journal.summary() self.messages = self.protocol.reset_messages( task=self.task, journal=journal_summary, reason=str(violation), ) # Reset per-loop tracking but keep the journal self.n_consecutive_format_errors = 0 ``` --- ## CLI Usage (standalone, no FestBench) ```bash # Install pip install zubble-agent # Run against any OpenAI-compatible endpoint zubble run \ --endpoint http://192.168.5.117:8000 \ --model nemotron-3.5-bolt-74b \ --task "fix the kubelet config on node-1" \ --env docker \ --env-container bastion-03 \ --level xhigh \ --minutes 15 \ --stop-sequences "User:" "\nUser:" \ --temperature 0 # Run against a hosted API zubble run \ --model anthropic/claude-sonnet-4 \ --task "refactor the auth module" \ --env local \ --protocol swe # Run with trial journal + cross-run memory zubble run \ --task "debug the cert_expiry_triage scenario" \ --env kubectl \ --memory-dir ~/.zubble/memory \ --journal # Run as a Ralph Wiggum loop (fresh context each iteration) zubble loop \ --task spec.md \ --max-iterations 20 \ --fresh-context \ --completion-signal "" ``` --- ## Integration with FestBench `zubble` is designed to run independently, but FestBench can use it as a drop-in replacement for `scenariobench.py`'s `make_backend`: ```python # In scenariobench.py (future integration) from zubble import Agent, OpenAIModel, DockerEnv, RepairProtocol, Journal def make_backend(via, model, base, container=None): if via == "zubble": env = DockerEnv(container=container) mdl = OpenAIModel(endpoint=base, model=model) protocol = RepairProtocol(report_path="/findings/report.md") journal = Journal(work_dir=f"/tmp/festbench_{container}") def gen(messages, timeout, kwargs=None, maxtok=4000, stop=None): agent = Agent(mdl, env, protocol, journal, level_kwargs=kwargs) result = agent.run_task(messages, timeout=timeout, maxtok=maxtok) return result.text, None, result.perf return gen ``` FestBench's scenario orchestration (kind clusters, oracles, CWD tracking) stays in `scenariobench.py`. `zubble` handles only the agent loop — the part that was broken. --- ## Comparison The four harnesses below cover the corners of the industry design space `zubble` distances itself from. **OpenHands** is the heavyweight multi-agent platform — web UI, Agent-Client Protocol, Docker sandboxes, automations, and a multi-repo TypeScript/Python codebase. **Aider** is the terminal pair-programmer: git-integrated, litellm-backed, with an architect/editor split for reasoning-then-edit workflows. **opencode** is the permission-driven CLI agent — MCP support, subagents, and a built-in `doom_loop` permission that triggers when the same tool call repeats three times. **pi.dev** is a minimal, extensible TypeScript harness with skills, extensions, packages, MCP and ACP support; it is also the harness FestBench currently runs via `pi-harness:0.85.1`, so the `pi.dev` column shows exactly what `zubble` adds beyond the status quo. | Feature | mini-swe-agent | Ralph Wiggum | ZeroStack | OpenHands | Aider | opencode | pi.dev | **zubble** | |---|---|---|---|---|---|---|---|---| | Core LoC | ~200 | ~400 (bash) | ~30K (Rust) | ~100K+ (TS/Py) | ~50K (Py) | ~100K (TS) | ~50K (TS) | **~200 (Py)** | | License | MIT | MIT | GPL-3.0 | MIT | Apache-2.0 | MIT | MIT | **MIT** | | Stop sequences | partial | N/A | no | no | no | no | no | **yes** | | In-stream repetition detection | no | no | no | no | no | no | no | **yes** | | Proper messages array | yes | N/A | yes | yes | yes | yes | yes | **yes** | | Doom-loop detection (3x) | no | no | yes | partial | partial | yes (built-in `doom_loop` permission) | no | **yes** | | Fresh-context reset | no | yes (core) | yes | no | no | partial (compaction) | partial (auto-compaction + branch summary) | **yes** | | Trial journal | no | no | no | no | no (git commits) | no (snapshots) | no (sessions JSONL is telemetry, not fed back) | **yes** | | Cross-run memory | no | no | no | no | no | yes (memories + skills) | yes (skills, extensions, packages) | **yes** (optional) | | Thinking level control | partial | N/A | no | yes | yes | no | yes (per-model overrides) | **yes** | | Per-call maxtok by level | no | N/A | no | no | no | no | no | **yes** | | Wall-time / step / cost limits | yes | yes | yes | yes | partial | no | partial (overflow recovery, context budget) | **yes** | | Stateless execution | yes | N/A | yes | no | no | no | no | **yes** | | Sandbox isolation | partial | N/A | yes (bubblewrap) | yes (Docker) | no (host fs) | partial (permissions) | no (no built-in sandbox) | **yes** | | Pluggable environments | yes | no | no | yes (backends) | no | no | no | **yes** | | Pluggable models | yes (litellm) | no (CLIs only) | yes | yes | yes (litellm) | yes (providers) | yes (providers, llama.cpp, custom) | **yes** | | FestBench integration | no | no | no | no | no | no | yes (`pi-harness:0.85.1`, `--via pi`) | **yes** (optional) | | Standalone operation | yes | yes | yes | yes | yes | yes | yes | **yes** | | GUI / TUI | CLI | CLI | CLI | Web UI | TUI | TUI + desktop | TUI (+ print/JSON/RPC/SDK) | CLI | | MCP / ACP support | no | no | no | ACP | no | MCP | both | no | | Multi-agent orchestration | no | no | no | yes (automations + agent server) | partial (architect/editor split) | yes (subagents, `subagent_depth`) | no (explicit; skips sub-agents and plan mode) | no | --- ## Roadmap ### Phase 0: Skeleton (this document) - [x] `zubble/` directory created in llm_benchmark repo - [x] LICENSE (MIT) - [x] DESIGN.md (this file) ### Phase 1: Structured protocol + plan-state ledger - [x] `protocol.py` — JSON schema, system prompt template, parse-retry wrapper - [x] `ledger.py` — plan state machine, attempted actions, abandoned approaches with causal failure reasons, divergence detection, summary injection - [x] `stream_chat` in benchlib.py — accepts optional `system_prompt` and `response_format` (defaults None; existing callers unaffected) - [x] `scenariobench.py` integration — `--structured` flag enables JSON protocol + ledger injection; existing text protocol preserved as default - [ ] Tests: re-run 5 xhigh DepthSweep scenarios, compare against 7303-rep baseline ### Phase 2: In-stream repetition detection - [ ] `repetition.py` — rolling 10-gram CRC32 fingerprint, last-200-token window - [ ] Graduated thresholds: 50 reps → soft nudge injected into stream 500 reps → parallel self-check (diagnostic, no action) 1000 reps → "STOP! Restart with fresh context or you will be terminated." (warning only, does NOT kill) 1500 reps → self-confirmation gate: model reviews progress, answers true/false. TRUE → kill. FALSE → continue. 3000 reps → unconditional kill (final safety net) - [ ] Self-check failure/timeout → continue, not kill - [ ] Tests for repetition detection at each threshold ### Phase 3: Ledger hierarchical summarization - [ ] Category buckets (network/compute/storage/auth/rbac/cert) - [ ] Collapse oldest entries into summary when `attempted` > 20 - [ ] Context reset with ledger summary on 1500-rep kill ### Phase 4: CLI + Polish - [ ] `run.py` — CLI entry point - [ ] `pyproject.toml` — pip installable - [ ] `README.md` with quickstart - [ ] Integration test: run one FestBench scenario end-to-end ### Phase 5 (deferred): Web search - [ ] Per-scenario opt-in flag in ScenarioBench YAML - [ ] DuckDuckGo/Brave/SearXNG fetch + model-extracted relevant steps - [ ] Measure search+apply vs pure-reasoning separately --- ## Attribution `zubble` draws heavily from: - [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent) by Princeton/Stanford (MIT) - [Ralph Wiggum](https://github.com/fstandhartinger/ralph-wiggum) by fstandhartinger (MIT), based on Geoffrey Huntley's methodology - [ZeroStack](https://github.com/gi-dellav/zerostack) by gi-dellav (GPL-3.0) — doom-loop and compaction patterns - FestBench's `scenariobench.py` — the hard-won lessons from degenerate loops