Agentic Programming in practice: building a harness for BioMysteryBench

Three things stayed with me.

1. Agentic programming is powerful because it moves your attention up the stack. The scarce resource stops being your ability to write code and becomes your ability to choose problems, define "done," and notice when a shortcut won't generalise. I spent almost none of this project reading diffs line-by-line and almost all of it deciding what to build and why and then reviewing trajectories to devise methods to improve the solution.

2. The harness plays as much of a role as the model — often more. This is not just my anecdote; it is the explicit thesis of the Anthropic engineering article that started me down this path [1], and my own runs pushed me to the same place from the opposite direction. Held to the same five problems, a cheap open model climbed from solving nothing on the first attempt (pass@1 0%) to 40%, and ended up solving every human-solvable problem — without the model itself getting any smarter — purely through harness fixes; on that solvable subset it finished level with Claude Sonnet at pass@5. The most convincing single data point was a problem that succeeded on one attempt only because that attempt drew a working download URL. The model mattered; the harness moved the needle at least as much.

3. Skills are how you direct an LLM and reduce its entropy. A tool hard-codes one implementation; a SKILL is a recipe the model reads and adapts. Providing a validated starting point doesn't force a solution — the agent can ignore it — but it narrows the search space and raises the floor, systematically shifting the distribution of outcomes toward correct answers. This is the same reason experienced scientists reach for established pipelines instead of deriving everything from first principles: the pipeline encodes prior knowledge about what works, constraining the practitioner toward productive solutions without removing their judgement. (One corollary I had to enforce repeatedly: SKILLs must not smuggle in question-specific hints — naming the gene or transcription factor for a given problem inflates benchmark scores but doesn't generalise, so it's banned.)

Put those three together and a bigger picture emerges. We can think of the LLM as non-deterministic middleware: a stochastic layer whose job is to reliably direct intent through deterministic tools, with SKILLs acting as the guide-rails that make the routing dependable. The BLAST saga is the whole thesis in miniature — the "intelligence" was never really the bottleneck; the reliability lived in whether a deterministic tool returned a legible result, and SKILLs were how I taught the stochastic layer to route through it correctly.

This framing is not entirely original to me; it is closely aligned with a central design pattern in neuro-symbolic AI, where probabilistic interpretation and planning are separated from deterministic, reproducible and verifiable execution [16][17]. VADAOrchestra, for example, uses an LLM to select and adapt workflows while dependency graphs, tools and a symbolic engine enforce execution order and perform exact inference. SPL makes the division explicit in its language, routing generative work to an LLM and exact or verifiable work to deterministic kernels.

What may be under-appreciated—and what my experience with BLAST brought home is how much of that reliability can reside in mundane harness engineering rather than in a formal logic engine: the features file, tests, SKILLs, accumulated learnings and even the return-code table for a forty-year-old command-line tool. The model still supplies semantic flexibility, but the surrounding contracts determine whether that flexibility can be translated into a reproducible result.

A practical heuristic falls out of this: push determinism outward. Every time I could move a decision from the model into a deterministic tool or a written-down SKILL — a return-code table, a query-size rule, a fixed download recipe — the system got more reliable and cheaper, and the model got one fewer opportunity to be creative in a place where I did not want creativity. What is left for the model is the part that genuinely is ambiguous: which strategy fits this problem, how to read a surprising result, when to change course. Seen this way, much of what people call "prompt engineering" is really environment engineering that steers local context — you are not so much coaxing a cleverer answer out of the model as arranging the world around it so that the obvious next step is also the right one. This idea is supported by a recent review paper on harness engineering [31].

I’ll go one step further, with an analogy for the molecular biologists in the crowd. A cell responds to a signal by changing the abundance, localisation or post-translational state of transcription factors. Those factors bind cis-regulatory elements whose sequence, arrangement and chromatin context determine which combinations of signals can activate a gene. Bound transcription factors can then recruit co-regulators, Mediator, general transcription factors and RNA polymerase II, increasing the probability of productive transcriptional initiation.

In the analogy, the cellular signal is intent; the transcription-factor network is the stochastic machinery that interprets it; the cis-regulatory grammar is the YAML front matter of a SKILL, determining when a capability becomes accessible; transcription initiation is the tool call in that SKILL; the coding sequence is the deterministic code for the stored implementation; and the expressed protein or enzyme is the running tool.[a][b]

The molecular events are not deterministic. Transcription-factor binding fluctuates, promoters fire in bursts and individual molecules behave stochastically. Yet cells produce remarkably reliable outcomes by controlling concentrations, affinities, accessibility, compartmentalisation and feedback—arranging the molecular environment so that the appropriate next event becomes the probable one.

Life and AI both work by modifying local context.

The most uncomfortable lesson of the project is already in Part 1, in four words I wrote to myself after running three agents in parallel: "I'm the bottleneck." The agents were fast and cheap; what could not be parallelised was me — reviewing plans, approving pull requests, deciding what "done" meant. Three agents produced three branches and three conversations faster than one human could meaningfully supervise, and the speed-up evaporated into context-switching.

This is worth sitting with, because it inverts the usual anxiety. The scarce resource in agentic development is not model capability or compute; both are abundant and getting cheaper. The scarce resource is human judgement applied at the right moments — the review, the constraint, the "no, that won't generalise." Scaling this kind of work does not mean adding agents until the human drowns; it means making each human decision higher-leverage — better plans to approve, thinner slices to review, better instrumentation so that a glance at a table tells you what a thousand log lines would. The analyze-trajectories SKILL earned its keep for exactly this reason: it was never for the agent, it was for me, so that my scarce attention could land where it counted.

There is a version of the dark-factory future (Part 1) in which the human reviews nothing and only judges outcomes. Perhaps. But everything I saw suggests the near-term win is not removing the human — it is spending the human's attention wisely, and building the harness so that spending it is cheap.

A stateless model has no memory of yesterday. Every session starts cold; the model that fixed a bug on Tuesday cannot recall it on Wednesday. And yet the project visibly learned over its 60 hour development time — it stopped repeating mistakes, accumulated conventions, and got harder to knock off course. That memory does not live in the model. It lives in the harness: code-learnings (a numbered list of lessons, each tied to the bug that taught it), session-close (which reconciles the documentation at the end of every session), features.md, the code walkthroughs. The model is amnesiac; the scaffolding remembers on its behalf, and CLAUDE.md is the note it reads back to itself each morning.[c]

I think this is the most under-appreciated part of building with agents, and it reframes what you are actually doing. You are not just getting code written. You are building a small institution — with rules, a memory, and an onboarding document — whose only employee forgets everything each night. A surprising amount of my highest-leverage work was institutional design: deciding what deserved to become a durable lesson, what belonged in a SKILL, and what the session-close ritual should enforce.

It also explains a temptation I had to keep resisting: anthropomorphising the agent. When the bioinformatics agent developed "time-angst" — abandoning good strategies because it seemed rushed by a clock that did not exist — it was tempting to read as nerves. The lens is useful right up until it isn't: the fix was not reassurance but a line of text telling it, plainly, that there is no time limit. The agent has no feelings to soothe; it has a context window and a prompt, and the harness is where you edit both.

If I carry one idea forward, it is this. The failures in Part 2 were overwhelmingly about tools — a BLAST call the agent could not size, a genome download it could not do reliably, a SKILL it never thought to read. Each time, the fix was for me to notice, diagnose, and write a better SKILL. What if the harness did more of that itself?

The sketch — which I have not built yet — is a self-hardening loop. The agent is told not to assume it knows how to install or use a tool unless that knowledge already lives in a SKILL; when it doesn't, it asks for help (a web search, the documentation), establishes a working method, and then writes that method into a new SKILL for the agents that follow. When a SKILL later fails, the agent investigates, derives a fix, and hardens the SKILL rather than working around it once. Memory that today I curate by hand would, in this picture, accrete on its own — each problem leaving the harness a little more capable than it found it.

That is speculative, and the failure modes are obvious (a SKILL that hardens in the wrong direction, drift, the need for a human gate on what gets written down). There is also some concern regarding generalization. SKILLs, written unskillfully will solve an individual problem well but may not work as well for others; the system becomes brittle. Despite the challenges, it is the natural extension of the whole thesis: if reliability lives in the scaffolding rather than the model, then the highest-leverage thing you can build is scaffolding that improves itself.

I'll note, with some amusement, that Anthropic appears to agree. A couple of months after I finished this work, they shipped Claude Science — an AI workbench that wraps the same Claude models in a purpose-built environment connected to sixty-plus scientific databases and field-specific toolkits [30]. Its explicit bet, as widely reported, is that better workflows beat better models. That is the argument of this entire series, now packaged as a product — which I find more encouraging than threatening: it means the lever I spent 60 hours discovering on a 2020 laptop is real enough to build on.

This has been a description of my current practice of agentic AI coding. It deliberately does not address questions like: should you be learning agentic programming? Should we be using it at all? Will AI replace bioinformaticians? These are valid, but they are better addressed once one has a non-populist, first-hand account of what we are all actually talking about. This series is not a tutorial, advice, a literature review, or recommended practice — it is an accounting of one human's experience in interesting times. My hope is that people reach out to share their experience, criticise, and offer advice.

a. I'm a molecular biologist, so this analogy came easily. After writing it I went looking for whether the parallel had already been drawn; the closest I found, from this month, is Banu [32], which maps promoters to schemas/context windows and genes to agent capabilities.

b. The parallel between transcriptional regulation and computation was drawn well before 2022 — see Istrail and Davidson [33][34], Buchler, Gerland and Hwa [35], and, on the biophysical side, Sabari et al. [36].

c. The film Memento (2000) is the apt pop-culture parallel: its protagonist, unable to form new memories, offloads state to notes, Polaroids and tattoos — an external harness for a mind that resets each day [37].