Agentic Programming in practice: building a harness for BioMysteryBench

Part 1 ended with a sound but unremarkable harness running Claude Sonnet. Part 2 is about deliberately making my own life harder — swapping the frontier model for a weaker open one — and what that taught me about where performance actually comes from. But first, the insight that made the swap possible at all.

Early on I asked Claude a precise question: when we call the Claude API, are we getting a direct response — just prompt in, response out — or is something else (reflection, server-side tools, SKILLs) happening before the answer comes back? The answer was unambiguous: our calls are direct. We use client.messages.create() with a custom bash tool definition, and the API returns either text or a tool_use block, nothing more. The entire ReAct loop — "reason, act, observe, repeat" — is implemented on our side in harness/agent.py. Each API call is a single forward pass; we execute the tool locally in Docker, append the result to the message history, and call again. The "agent" behaviour emerges from our orchestration, not from anything the API does autonomously.

The consequence is liberating, and it is the through-line of this whole part: memory, loops, critics, prompt injections, and multi-agent coordination are all ours to build, and they are provider-agnostic by design, because they operate on the message history and tool results rather than on anything internal to the model. The only provider-specific wrinkle is the format of a tool call — Anthropic's tool_use/tool_result content blocks versus the OpenAI-style function/tool_calls used by most open models. A thin adapter layer (harness/llm.py) normalises that difference so agent.py never sees it. Once that adapter existed, pointing the harness at a different LLM was a command-line flag, not a rewrite.

Portability was a goal for this project, and it is worth being precise about what the costs are.

Two of the relevant pieces are open — as specifications. The Model Context Protocol (MCP) [15] — Anthropic's open standard for connecting models to tools, donated to the Linux Foundation's Agentic AI Foundation in December 2025 [27] — can be spoken by any tool-calling model. And Agent Skills, the SKILL.md format for packaging a reusable recipe (a Markdown file with a little YAML frontmatter, optionally bundling scripts and references), were released as an open standard the same month [25]. The format is now read by Claude Code, Cursor, Codex CLI, Gemini CLI, and local open-source models alike.

But an open format does not make a given SKILL portable. A SKILL is portable only to the extent that the things it calls are. A SKILL.md is just a recipe; if its steps invoke a proprietary, Anthropic-hosted MCP server or a Claude.ai connector, then running that SKILL needs Anthropic's infrastructure and an Anthropic key, and it will not work under another model no matter how open the surrounding format is. The portability lives in the dependencies, not the wrapper — a point even Anthropic's own documentation concedes, noting that SKILL portability "depends on the implementation" and that vendors increasingly ship an open SKILL alongside their MCP connector (Canva, Notion, Sentry) so the two travel together [26]. This coupling is the real lock-in vector, and it has been flagged as such in commentary on the SKILLs/MCP ecosystem [28][29]. In many cases, the MCP connector may itself be open code making it easier to modify or replace. In other cases, it may be closed and you may not care. The point is to be aware of the dependency ahead of time.

Taking a look at Anthropic's own public anthropics/skills repository, you can see examples of both:

The format and the protocol are open, and the majority of Anthropic's published SKILLs are genuinely model-agnostic Python. But the moment a SKILL provided by a third party reaches for a proprietary connector, you are back on the hook for dependency on that provider's stack. For this project I want to sidestep the whole question by writing my own SKILL.md files that call only open Python and open MCP, so they run unchanged under gpt-oss on Cerebras — which is exactly the cost of portability this section is about.

Going provider-agnostic doesn't cost you the ability to use SKILLs or MCP — it costs you the turnkey, managed integration. If Anthropic were to provide a custom set of SKILLs that called their own proprietary MCP servers that say carried out bioinformatics tasks, using that stack might wed you to it permanently. For some users, having a managed stack of SKILLs that have a tested, versioned, and hosted set of MCP servers behind them might be exactly the type of product they want. But for a project whose entire purpose is to test a harness across models, you need the option to control dependencies - the section below addresses this issue and the codebase shows how it can be accomplished.

I hasten to add that Anthropic's strategy does not appear to reply on vendor lock-in. Anthropic's open-sourcing of the tooling layer (MCP, the SKILLs format) was intentionally meant to prevent lock-in and avoid duplication of effort. And their recently released Claude Science(see my review) product leans heavily on integrating open source, public databases and resources. Instead, their moat is elsewhere — model quality, safety and trust for regulated buyers, an integrated end-to-end stack, and the usage flywheel. Publishing a benchmark like BioMysteryBench is part of that argument: our model does this; can yours? Commoditising the plumbing while competing on the model is good strategy for Anthropic and for the many users who want the managed path. It just isn't the path for this harness experiment.

Once the harness worked with Claude Sonnet, I switched the agent model to an open one. Three reasons:

That third reason could provide a strategy for long-term development of this project. Rather than reach for the best model we can afford, deliberately use a mediocre one. This means that any realised gains belong to the development carried out and are not simply rented from the frontier model it rests on.

Getting a capable open model to actually run was more of an adventure than I expected, and the sequence is instructive because every dead end was a hardware or economics wall, not a code wall.

Local, on my laptop — a non-starter. The tidy dream was to run a model like qwen2.5:14b locally under Ollama. It failed on physics: my 2020 Intel Mac has an Iris Plus integrated GPU with about 1.5 GB of shared VRAM, and the 4-bit quantised weights need roughly 9 GB. Ollama fell back to CPU and produced perhaps two tokens per second — unusable for an agent that makes dozens of calls per problem. (On an Apple-silicon Mac with unified memory this would have been fine; on my machine it was hopeless.)

Groq — fast, then blocked. Groq runs models on custom Language Processing Unit (LPU) hardware at genuinely high speed (hundreds of tokens per second). It worked, briefly. But its free "Dev Tier" was blocked during my window and I ran out of credits, so it stopped being an option through no fault of the code.

Cerebras — the workhorse. Cerebras runs open models on wafer-scale hardware at up to ~2,000 tokens per second, with a 1-million-token-per-day free tier and an OpenAI-compatible API — so the adapter layer meant zero code changes to adopt it. I settled on qwen-3-235b-a22b-instruct-2507: a 235-billion-parameter mixture-of-experts (MoE) model with only ~22 billion parameters active per token, chosen for its tool-use ability and its speed. This became the open workhorse for most of the project.

The moral of the odyssey: for a laptop-based developer in 2026, "open model" almost always means "open model hosted somewhere with real accelerators," and the binding constraints are free-tier limits and provider stability, not your ability to write the integration.

The first substantial harness feature beyond the basic loop was a critic — a second LLM call, with its own system prompt (often a cheaper model such as Claude Haiku acting as judge/critic), that reviews the agent's trajectory before it commits to a final answer. In principle it is exactly the kind of thing that is "ours to build": it lives entirely in the orchestration layer, independent of the API.

In practice, the critic was where I learned the hard lessons about how to build with an agent. A cluster of bugs surfaced in quick succession: the critic was constructed but _run_critic() still called the main agent's client instead of the critic's (so critiques silently went to the wrong endpoint); the same cross-provider routing bug independently existed in the scorer's judge; a deprecated Haiku model name got "corrected" in the wrong direction and had to be reversed a PR later; and an empty system prompt tripped a cache_control 400 error from the Anthropic API that only fired when a secondary client with no system prompt was used. Several of these were invisible because an except Exception: return "" swallowed the failure — the run continued, silently degraded.

Two durable practices came out of this, and they are the reason the rest of Part 2 went more smoothly than it might have:

The critic itself eventually earned its place — but only after I could see what it was actually doing, which required the next piece.

You cannot improve what you cannot see. The most valuable tool I built was not part of the agent at all — it was an analyze-trajectories SKILL that turns a directory of raw run logs into a single wide table, one row per attempt and 33 columns of extracted detail: steps taken, wall-clock time, cost, how many rate-limit back-offs fired, which tools the agent claimed were missing (BLAST? python3? bedtools?), what the critic said and how the agent responded, the raw and cleaned final answers, whether the judge scored it correct — and, crucially, a column for my own read of whether the answer was really correct. It was built the way everything else was: twelve sub-slices, 53 unit tests, deterministic extractors for the mechanical fields and a mocked LLM for the interpretive ones.

Reading those tables changed how I thought about the whole project. In failure after failure, the model had made a locally rational decision given wrong or missing information: an empty BLAST result mislabelled as "no hits"; a non-existent time limit it kept reacting to; reference recipes it never consulted because they weren't actually in the container image it was running. The diagnostic question I kept returning to was: would a competent domain expert, given exactly this context, make the same mistake? When the answer was yes — and it usually was — the fault lay in the harness, not the model. The tables didn't just diagnose individual runs; they pointed, over and over, at a small number of environmental defects. Fixing those is the rest of this part.

If I had to nominate a single most-important lever in the entire project, it would not be a model, a prompt, or the critic. It would be the humble matter of whether BLAST — the Basic Local Alignment Search Tool, the workhorse for identifying a biological sequence by comparing it against a database — actually worked. The story is worth telling in full because it is a near-perfect case study in harness-over-model.

It began as a mystery: on the bacterial-identification problem (hb002), the agent kept concluding that BLAST "wasn't installed" and burning a dozen steps trying to reinstall it. I assumed a reasoning error. It was worse than that — a stack of distinct defects, each hiding behind the last:

The fixes were unglamorous and decisive. Add blast to the image and pre-flight-check its version before every run. Map every BLAST return code to a distinct, accurate message — timeout, network stall, rate-limit, out-of-memory, genuine no-hits — so the agent can never again read "the tool failed" as "the organism isn't there." Change the default output format to include species names (sscinames) so the answer is legible without a memory lookup. And add explicit query-size guidance to the BLAST SKILL.md (never send the whole genome; extract a short region) — while scrubbing that same SKILL of any problem-specific hint, since encoding "use 16S for this one" would inflate the benchmark without generalising.

The payoff was stark. On 5 attempts at question hb002, the bacterial-ID problem, the working-BLAST harness took the result from 0 of 5 correct to 5 of 5 in the final run — with no change to the model's underlying intelligence at all.

Two more environmental defects deserve a mention, because both masqueraded as model failures and neither was.

The I/O deadlock. On long benchmark runs, some attempts began aborting with Resource deadlock avoided while merely reading the input data file. The cause wasn't the agent; it was that the read-only bind-mount I used to expose each problem's data flaked out during long-lived Docker Desktop sessions on macOS. Replacing the bind mount with a put_archive call — copying the data into the container once, up front — made it disappear. That single infrastructure change allowed attempts on two problems (hb022 and hb053) to run to completion.

The phantom clock. The agent had chronic "time-angst": it would announce that it was changing strategy "due to time constraints" and abandon a perfectly good approach — a genome download, say — when it hit the container's 10-minute per-command ceiling. But there is no wall-clock budget in the harness; there is only a step limit. A correctly formed download to the right mirror finished in under ten minutes (I confirmed one that completed in 9m45s); the agent was giving up on tasks that would have succeeded. The remedy was to tell the agent, explicitly, that there is no time limit — only a step limit — and to surface the current step count up front, exactly as I already surfaced its RAM and disk budget. A related lesson: the step "limit" was a soft nudge (a prompt fired once at the cap) rather than a hard stop, so on one run the agent sailed 18 steps past a maximum of 100. A nudge in a prompt is not an enforcement.

Midway through, Cerebras deprecated qwen-3-235b-a22b-instruct-2507 out from under me (removed 27 May 2026) and pointed everyone at gpt-oss-120b — a 120-billion-parameter open reasoning model, dense rather than MoE, running at ~3,000 tokens per second with a 131 K context window. The harness's adapter layer meant the switch itself was trivial. What wasn't trivial was that the new model appeared to have stopped thinking.

The trajectories looked hollow: bare tool call after bare tool call, with none of the reasoning narration the harness was designed to log. My first instinct was that the model was simply terse. It wasn't. gpt-oss-120b is a reasoning model that emits its chain-of-thought on a separate "analysis" channel of OpenAI's Harmony chat format [21] — while tool calls go out on a "commentary" channel and only user-facing text lands on the "final" channel. Our code read only the content (final) field, so on every tool-use step it logged an empty string. The model had been reasoning the whole time; we were reading the wrong channel.

The fix was almost insultingly small relative to how long the mystery had felt: read the msg.reasoning field (which Cerebras already returns by default in its "parsed" mode), and, while I was there, set reasoning_effort=high to buy more thinking tokens on the hard problems. Reasoning re-appeared in the logs, rendered as collapsible blocks in the Markdown trajectories, and hard-problem answers improved. The lesson went straight into code-learnings: "the model is being quiet" is sometimes a harness bug wearing a model costume — and, more concretely, always confirm you're using a model's native serialisation format the way it was intended, because gpt-oss "should not be used without the Harmony format."

So where did performance actually come from — the model, or the harness around it? I have real numbers now, all on the same five-problem preview set, and they tell a consistent story. The small 5 problem data set I developed against was composed of 3 human-solvable and 2 human-difficult problems. In the table, pass@1 is the fraction of problems solved on the first attempt and pass@5 the fraction solved on at least one of five attempts.

Two things stand out. First, the open model, held to the same five problems, climbed from solving nothing on the first attempt (pass@1 0%) to 40%, and ended up solving every human-solvable problem — 14 of 15 human-solvable attempts correct (93%) in the final run — without the model on the open side getting any smarter; the only reason it changed at all was that Cerebras retired one model and I adopted its recommended replacement. Every gain in between came from the harness: the rate-limit back-off, the BLAST return-code mapping, the I/O fix, the reasoning-visibility fix.

Second, and more pointedly: on the human-solvable subset, that cheap open model ended level with Claude Sonnet — both at pass@5 3/3. I want to be careful not to over-claim here. This does not show a weak model beating a strong one head-to-head; Sonnet still led on first-attempt reliability (60% pass@1 versus 40%), and the two human-difficult problems stayed unsolved in the end (though one intermediate pass by Qwen3 managed 2/5 on one hard problem (hb053) but wasn't reproduced). What it does show is that a large fraction of the distance between "solves nothing" and "solves everything a human could" was closed by fixing the environment, not by upgrading the intelligence inside it.

That is the empirical basis for the claim that opens Part 3. Reading the 33-column tables, the recurring pattern was a competent decision made on bad information — and the single most convincing data point was a problem that succeeded on exactly one attempt for the mundane reason that that attempt happened to hit a working download URL while the others hit slow mirrors. Same model, same prompt; the environment was the difference. At this stage of the project, harness fixes were higher-ROI than model upgrades — and it wasn't close.

Continue to Part 3 — Reflections