The Perfect-Model Test for Agent Harness Engineering

TLDR: Both camps in the model-versus-harness argument already agree on more than they admit - own your interfaces, invest in verification and context continuity, gate the steps that can hurt somebody. What they disagree about, without ever putting it this way, is where the line falls between compensation and control, and who is allowed to draw it. I think only the operator can draw it, and the test is the perfect-model test: would I still want this under a perfect model. This post names the overlap, names the three things control actually buys, makes the strongest case against me, and ends with five questions to run before your next agent feature ships.

The perfect-model test: build the part of the harness you would still want if the model were perfect. That part is control rather than compensation, and it does not depreciate. Everything else is a loan against a release schedule you do not control.

The same sentence keeps ending the same argument, and by now I cannot remember where I last heard it. A design review, where someone is working through what an agent feature is actually going to need and someone else says: let's not over-engineer the harness, the next model will do this natively. The room nods. It sounds like discipline - like somebody who learned the lesson of the last three years and is saving the team from building a museum.

Anyone serious about their product should be doing the opposite of what that advice says. A harness is your insurance against the gaps in whatever model you are renting this quarter, and that much is obvious. What gets missed is that it is also the only thing on your side of the line your competitor cannot rent - an agent is only ever as good as the harness carrying it, and the harness is the part you own. Carlos Rymer held Gemini 3.6 Flash fixed and swapped the scaffold around it: across eight tasks, resolution ran from 63.6% down to zero, a 63.6 point spread with nothing changing but the harness [7]. A small study, and a noise band wide enough that only the extremes separate - I will come back to that - but these are the extremes. Same model, same tasks, and the only variable was the thing everyone is telling you not to build. A spread that size reaches your customer long before it reaches a leaderboard - one of them got an answer and one of them got nothing.

It is popularly accepted that an agent is a model plus skills plus tools. Look at the first half of that right-hand side. A model with skills is still a model - it reasons, it produces, and then it stops. What makes it an agent is that it can reach out and act, and every part of that reaching is code somebody on my team wrote. The harness is that code: orchestration, tool invocation, context management, retries, state, resumption, the gates on anything that can hurt somebody - everything on my side of the API line. A more reliable harness is a more reliable agent, and there is no arrangement of those three terms that comes out the other way.

The parts of that harness worth building are the parts I would still want if the model were perfect. Not the retry shims and the prompt tricks that paper over what today's weights fumble - those really do depreciate, and the people who keep saying so are right about them. The durable parts were never compensation. They are what I need in place before a stochastic component goes anywhere near a paying customer, and a better component removes none of them.

And that is the only ceiling I care about. Not a number on a benchmark - the ceiling on what my service can promise and still keep.

"The next model will do this natively" is a bet on tomorrow. It might be a good bet. Tomorrow has been arriving roughly on schedule for three years now. But I am not running a service on tomorrow's weights. I am running it on today's, for customers who filed today's tickets.

And look at what that sentence is really asking for. Not that I build less. That I ship a production commitment against an untested assumption about next quarter's model, and leave the failure modes unnamed in the meantime, because naming them would mean building something.

Both camps already agree on more than they say

By the middle of this year, "the harness sets the ceiling" had stopped being a contrarian read - Addy Osmani's version, that a decent model with a great harness beats a great model with a bad harness, is by now a canon with a vendor category attached [1]. The argument still live is narrower and more interesting: is harness work an asset or an expense? Osmani says build the ratchet and keep it - every mistake the agent makes becomes a permanent configuration change. The other camp says most harness code is a wager against the next model generation, and the sharpest version of that is attributed to Boris Cherny, paraphrased rather than quoted: scaffolding buys you a modest single-generation gain, and the next generation wipes it out [2]. Lance Martin pushes further and says architectural assumptions expire in about six months, so build the harness to be deleted [3].

The cleanest statement of the premise underneath all of it comes from inside the harness camp. Anthropic's Managed Agents post argues for decoupling the harness from the execution environment on the explicit grounds that a harness encodes assumptions about what the model cannot do on its own, and that those assumptions go stale as models improve [4]. That reasoning has both sides of the line in it: the most institutionally credible voice for harness design, using the compensation premise to conclude the harness is temporary. The bitter-lesson camp starts from nearly the same premise and arrives at the same place from the opposite direction. Nobody is arguing with anybody. They are running the same accounting.

And on that accounting, they are right. If a harness exists to cover what the model cannot do, it is depreciating by construction, and the only open question is the schedule.

Read either of those posts to the end, though, and the exception is already carved out. The same Managed Agents post tells you to build to stable interfaces you own - harness, session, sandbox - rather than to a model generation. The bitter-lesson post that carries Cherny's claim tells you to spend nothing on orchestration logic and to invest in verification, context continuity and safety gates [2]. Those are, near enough, the things I am about to argue for. So I am not going to pretend the depreciation camp told anyone to fly blind. They didn't. Nobody in this argument is against logging your steps or owning your memory policy, and a piece that wins by quietly widening the word "harness" until it covers the part nobody was contesting has not won anything.

Here is what is actually in dispute. Both carve-outs are drawn the same way - by predicting what the model will absorb next. Verification survives because models are bad at checking themselves. Safety gates survive because you cannot hand a model the knife. That is a forecast about somebody else's roadmap, and it produces a generic list: the same three or four items for every product, because a lab writing the list cannot see your customers. Draw the line the other way and the list changes. Stop asking "will the next model absorb this?" and apply the perfect-model test instead: would I still want this if the model were perfect? The first question makes your build plan a derivative of a release schedule you do not control. The second one makes it a statement about what your service has promised, which is a thing you can actually be held to. Only the operator can answer it, and the honest answer is longer than three items.

Compensation depreciates. Control does not.

Hand me a model that can take a genuinely complex task and carry it end to end, correctly, today. I still want the harness. Not because I doubt the model - because I am running a service, and a service has to be able to say what it did and where it stopped. I want the task broken into steps I can log, replay, price and bound, in a structure I chose. That is not help for the model. That is a record for me. A demo gets to be one opaque success. A production request does not.

This is where the shared premise does its real damage. It makes the harness an apology for the model, which means every improvement in the model reads as an argument for less harness. But the things I most need a harness for were never deficits, and they do not soften as the component gets better. Compensation depreciates. Control does not.

Three things the harness buys that a better model does not

Visible decomposition. I want the plan before the output, in a shape my own code can read: steps, tool calls, intermediate state, a resumption point. When a request dies at step seven, I want to know it was step seven, and I want to restart from there rather than from the beginning. Everybody agrees with this, including the camp I am arguing with - which is the point. A better model does not make it cheaper, and nobody is waiting for the model to hand it over.

The knob your use case lives on. Every product has one aspect of model behaviour that carries most of its value, and no harness vendor ships yours. Run an AI personal assistant and memory management is the whole product - what gets remembered, for how long, under whose control, which fragment surfaces at the moment it matters and which one stays buried because surfacing it would be a small betrayal. None of that is a capability gap. A perfect model with a perfect memory would still not know that my users want the work context on a Monday morning and not the family one, because that is not a fact about intelligence. It is a fact about my customers, and the only place it can live is in code somebody on my team writes. This is also why a generic carve-out can never contain it: a lab's list of things worth building is the list that holds for everybody, and a knob is by definition the thing that holds only for me. Nicolas Bustamante makes the point harder than I would: models are post-trained against specific harnesses, with tool surfaces and citation conventions baked into the weights, and the labs ship matched pairs - Claude Code with Claude, Codex CLI with GPT - which he reads as the lab's moat [5]. Take that seriously and adopting a lab's harness wholesale means your differentiation lives in somebody else's weights, on somebody else's release schedule. A better model does not make this cheaper.

Attribution. A failure you cannot assign to a side is a failure you cannot fix, and assigning it is the part you cannot rent. Raj and co-authors are right that this field has a repair-assignment problem: identical-looking failures imply different fixes, so a failure has to be localized to an interaction edge with a named fault side [6]. What they do not say is that you cannot run their taxonomy on a system you do not own. Their edges are model to harness, model to tools, harness to environment, and assigning a fault on any of them means seeing both ends of it. Rent the harness, take its trace as given, and you are looking at one end and inferring the other. The trace you get back shows the request and the outcome. It does not show which retry swallowed the tool error, or what the context window looked like on the third attempt, or whether the plan was truncated before the model ever saw step seven. So every incident review lands on the same sentence - something in there went wrong - and the only move available is to file a ticket and wait for a release note. That is not a diagnosis, and worse, it does not accumulate. A team that owns the harness gets something out of each incident that outlasts it: after twenty of them they know the shape of their own failures, which model releases arrive with, which ones arrive with a config change, which ones only ever show up under load. That is the one asset in this entire argument that a model generation cannot wipe out. A better model does not make this cheaper.

Where this argument is weakest: build-versus-buy, and the Raj attribution rule

The objection I get most often is not academic. It is build-versus-buy, and so far I have not been answering it. Context management on overflow, malformed tool-call recovery, retries and rate limits, cancellation, sandboxing, injection defence at the tool boundary, tracing - that is months of unglamorous work, an off-the-shelf harness has already encoded most of it, and a team writing its own loop mostly rediscovers the same bugs a year later. If "own the harness" means "write all of that yourself," then this piece is asking for a quarter of engineering it never priced, and the objection lands.

But owning a harness is not the same as writing one. Own is not write, and that distinction is the whole answer. The three things above are properties of a system rather than a licence, and you can have all three inside a framework somebody else maintains. What you cannot do is adopt one and skip the question. The test is whether the answers come back in your terms: can your own code read the plan and the resumption point, or only a rendered trace after the fact? Is the one behaviour your product lives on a parameter you set, or a default someone else ships and can change in a minor version? When something fails, do you see both ends of the edge, or one end and a summary? Buy the loop. Own the interfaces the loop answers to, the knob, and the record. That is a much smaller bill, and it is the version of this argument I would defend in a planning meeting.

Raj and co-authors are the strongest published thing standing against me, and not for the obvious reason. They state their attribution rule out loud, which almost nobody else does: a failure is model-side if a more capable model could have prevented or recovered from it. Run that rule and 36 of their 41 failure modes land on the model. Take it seriously and most of what gets called harness engineering is waiting-for-the-model with extra steps.

I think that rule is the right thing to argue about and the wrong thing to design against. It settles credit after the fact using a counterfactual model, and I cannot deploy a counterfactual model. The practitioner canon runs the mirror-image rule - harness-side if a better harness could have prevented it - across the same incidents and gets the opposite answer, which should tell you the split is about accounting conventions rather than about the systems. My convention assigns the work to the team that can act this quarter, and I would rather defend that as a choice than pretend it fell out of the data.

The evidence here is also softer than the confidence in this genre suggests, mine included. Carlos Rymer's harbor-harness-spread is the most direct measurement anybody has published: hold Gemini 3.6 Flash fixed, swap the scaffold, and resolution runs from 63.6% down to 0% across eight scored tasks, a 63.6 point spread from scaffold alone [7]. The caveats I flagged at the top are his own and they are large: eight tasks after oracle screening, a compressed time budget, and a 16.7 point run-to-run swing on an unchanged harness, which means only the extremes are separable. deepset's engineering write-up reports harness-only changes producing substantial movement in Terminal-Bench rankings - their measurement, on their own system [8]. All of that measures how well a scaffold elicits a model on curated tasks. None of it measures how a system behaves under real load, with real data, and a real blast radius. The 74% rollback figure that circulates as proof of all this comes from Sinch's "AI Production Paradox" research, which asked 2,527 enterprise leaders what happened to their customer-communications agents [9]; the postmortem genre that carries the number treats it as a runtime finding [10], and it is a survey of decision-makers. I am not arguing from a measured production ceiling, because nobody has published one. I am arguing from what a service has to be able to do on a Tuesday.

And the depreciation camp is not wrong about the failure mode it fears. A harness that is a pile of workarounds for last year's model is waste, and worse than waste when the workarounds are load-bearing for each other. That is the trash can Mac Pro: a genuinely beautiful machine designed tightly around one thermal assumption, with nothing that could be swapped out when the assumption stopped holding. Do not build that. Build the harness so any single piece can be lifted out on its own the week the model absorbs its job, and so no piece is holding up three others.

How to decide what to build: five questions before your next agent feature ships

Before the next agent feature ships, get written answers from the team:

When a request fails halfway, what does the system know about where it stopped, and can it resume from there or only start over?

Which single aspect of model behaviour carries most of this product's value for our customers - and if we are not building the control surface for it, who is?

What is our rule for calling an incident a model problem or a harness problem, written down before the incident rather than argued about during it?

Which harness components would we delete if the next model absorbed their job, and can each of them come out on its own without taking the others with it?

If we skip a piece of this because we expect the next model to make it unnecessary, and that model arrives late or arrives different, what does the gap cost the customer - and is that a delay we can recover from, or a trust we cannot?

The heuristic is the perfect-model test: build the part of the harness you would still want if the model were perfect. That part is not compensation, and it will not depreciate. Everything else is a loan against a release schedule you do not control.

Back to the room, and the sentence that ends the argument. "Let's not over-engineer the harness" is a good instinct aimed at the wrong object. It hears every line of harness code as a bet that the model will stay weak, and on that reading restraint is obviously correct.

But the pieces worth building were never bets on the model. They are the conditions under which I am willing to hand a customer something a stochastic component produced: that I can see how it got there, that the one behaviour my product lives on is a knob I control, and that when it breaks I can say which half broke. None of that is a prediction about next quarter. That is the shape of a service.

So sure, do not over-engineer it. That is a real design question about how to build while capability rises underneath you, and the answer is modularity, a short list, and a clear line between what you buy and what you own - not abstinence. Compensation depreciates. Control does not.

References

[1] Addy Osmani, "Agent Harness Engineering," April 2026

[2] "The Bitter Lesson of Agentic Coding," April 2026, paraphrasing Boris Cherny

[3] Lance Martin with Hugo Bowne-Anderson, "AI Agent Harness, 3 Principles for Context Engineering, and the Bitter Lesson Revisited," 2026

[4] Anthropic engineering, "Scaling Managed Agents: Decoupling the brain from the hands," April 2026

[5] Nicolas Bustamante, "Model-Harness-Fit," May 2026

[6] Raj, Gupta, Mahmoud, Dumitru, Yi, Sabharwal and He, "Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures," July 2026

[7] Carlos Rymer, harbor-harness-spread, 2026

[8] deepset, "Harness Engineering: How to Build Reliable AI Agents by Engineering the System, Not the Model," May 2026

[9] Sinch, "AI Production Paradox" research, 2,527 enterprise leaders, 2026

[10] Jaroslaw Wasowski, "Why 74% of AI Agents Get Rolled Back in Production: A Runtime Post-Mortem," June 2026

The observations here are my own and don't represent Microsoft's views or positions. This post was written with the assistance of AI. The observations, experiences, and judgments are my own.