200 OK and Wrong: Silent Failures in AI Systems

Imagine a product retro is going well. Every panel on the board is green - latency inside budget, error rate at four nines, token spend flat for the third week running. Someone calls the launch boring and means it as a compliment.

Then the support lead turns her laptop around. Their assistant has told a customer, twice, in warm and fluent English, that a refund has been processed. It has not. There is no refund, and no row anywhere showing one was ever started. The model said the true-sounding thing and moved on, and for over a week nobody upstream of that customer knew. This isn’t a far fetched imagination. Product teams across the world are living a version of it almost everyday.

The board stayed green the whole time. It was built to.

So someone asks the reasonable question - what do we add to the dashboard so this shows up next time - and there is nothing to add, because no panel was ever going to catch it. Not because the problem is hard. The model can be wrong fluently, but it is almost never wrong invisibly: the wrongness lands somewhere physical, and here it landed in the gap between what the assistant asserted and what the ledger held. The team was watching the model. You do not start at the model and reason outward to what should be watched - you start from what the product is supposed to produce, and the measures fall out of that.

The next two weeks go to shopping for tracing vendors. They are the wrong two weeks.

TLDR: There is no universal observability pattern for LLM systems coming, and the reason is not that the problem is hard. It is that observability is being run as an engineering question about the model when it is a product question about the outcome. Define what the system is supposed to produce and the place a wrong answer lands - a ledger row, a reopened ticket, a request rephrased a third time - is where you instrument, not the model that produced it; search ran a harder version of this for twenty years without ever measuring the thing it cared about. Where the move stops is where the outcome is too diffuse for anything a user does to bound it, and there the honest answer is a sampled human read and a hard gate on anything irreversible, not a metric.

The literature studies the model alone

Everyone writing on this takes the LLM system as the unit of analysis. The output is the thing examined and the only open question is which instrument reads it. Bibin Kuruvilla, who is very good on the problem, will tell you plainly that an LLM application can return a fast, valid, well-formatted response that is completely wrong and no conventional APM metric will register it - and his answer is to alert on evaluation scores instead of status codes [4]. Dexing Liu, from the sharp academic end, says the silence is intrinsic, entropy accumulating on a clock, so reliability has to come from deterministic gates enforced outside the probabilistic path [2]. About as far apart as this literature goes, and both of them are reading the model.

The survey the genre cites reads the same way. LangChain's State of Agent Engineering - a June 2026 survey of 1,340 teams that Kuruvilla's post quotes - finds 89% of teams running AI observability and 29.5% running no evaluation at all, with 52.4% on offline evals and 37.3% on online ones [4]. The genre reads that as under-adoption of scoring. Read it the other way and it says a large group of teams instrumented before they knew what they were looking for. That is exactly what you do when you believe the signal lives inside the model.

César Migueláñez gives the position away in a sentence: agents return responses that look correct until downstream consequences reveal the error [5]. Downstream is a place. It has owners, tables, and usually a metric that predates the model by years.

John Hodge reports that the OpenTelemetry GenAI conventions still have nothing marked stable, with real frameworks emitting several attribute generations at once [3] - which everyone reads as a roadmap item. There is no stable attribute for "the answer was wrong" because wrong is a property of the outcome, not of the span, and a field cannot standardize a signal whose meaning is set one product at a time.

Relevance was never directly observable either

No search engine has ever known what fraction of its results were relevant. There is no counter for it. What search teams did instead was measure where relevance touches a person: which result got the click, how long it took to get there, whether the query got reformulated, whether the session ended or kept going. None of those is relevance. Positioned together, around the thing, they bound it well enough to run a business on.

The difference is that nobody on those teams expected the ranker to emit a relevance metric. They expected the product to define one.

An LLM system does not sit in a vacuum either, and it is doing one of two jobs.

Optimizing something you already do means a baseline of ideal already exists, and so, usually, do the measures. A customer support agent should be carrying the metrics you would have built for a person doing that job: resolution rate, escalation rate, repeat contact inside seven days, refunds asserted versus refunds present in the ledger. The question it keeps asking: what did we measure about the human step, and who owns that measure now?

Enabling something that was not possible before means there is no inherited baseline, and the question stops being an engineering one. It goes to the product manager, and it is: what are we enabling? If nobody can say in one sentence what the system is supposed to produce, no tracing layer will supply the answer, and every eval written before that sentence exists is guessing. The question it keeps asking: what outcome would a person outside this team recognise as this thing working?

The humans were the monitor

Wu's data is the sharpest thing in the literature and I think he draws the wrong conclusion from it. Before the numbers, make a guess: in a production agent runtime carrying 4,286 automated tests and 827 governance checks, what share of the silent failures did that suite stop before a user saw them? Zero.

Across the fifteen incidents Wu put through an ex-ante and regression audit, those 4,286 tests and 827 governance checks delivered zero percent ex-ante prevention and 87% regression blocking. Across his full corpus of twenty-two incidents with root-cause postmortems, roughly 70% of the silent failures were caught by a human noticing [1]. Failure lifetimes ran from 13 hours to 60 days, and the longest-lived ones sat in the seams between components where no test runs.

Both halves of that live in one runtime, which is why it is worth sitting with. The suite did precisely the job a suite is good at, holding a known failure down once it is known. It was also handed a job it was never going to do, which is predicting an unknown one. The failure was in the handing, not the suite.

And that 70% is not the absence of a monitor. It is a monitor nobody wrote down. A person noticed. In production that person is almost always a customer, and customers noticing is a signal with a rate, a latency and an owner. Wu's answer is to promote it into a weekly thirty minute human observation ritual. I would rather instrument it: contact rate on conversations the assistant marked resolved, reopen rate, the share of sessions where someone rephrases the same request a third time. Those are product metrics. They were never observability's to build.

Where the move stops

The real hole in this has two halves, and they are not the same hole.

The first is the genuinely new product. If you are enabling something nobody did before, there is no human funnel to inherit and no ledger to reconcile against. Worse, even where a baseline does exist, the first instance of a new failure class is unmonitored by definition, because you write the monitor after you learn the class exists. Liu's position covers that case and mine does not: a deterministic gate outside the probabilistic path does not need to know what good looks like, only what is forbidden [2].

Liu isn't wrong. Gates are cheap, they compose, and I would put them on anything irreversible. But as the governing principle it caps the system at whatever you could enumerate as forbidden on day one, which is a strange ceiling to accept for a technology you adopted because you could not enumerate the space in advance.

The second half is the one a practitioner actually raises, and it has nothing to do with novelty. A product can be ten years old and still have nothing to reconcile against. A summarizer, a research assistant, a drafting tool: human predecessors going back decades, a baseline of sorts, and no row anywhere that can contradict the output. A refund either exists in the ledger or it does not. A summary is not wrong in that way. If this argument only holds where the outcome is a database row, then it is an argument about transactional agents and it should say so.

It is not, and the search case is why. Relevance had no counter either, and the answer was never to find one - it was to position proxies around the thing until they bounded it. The same move is available here. For a research assistant: did the passage make it into the work, did the user go back to the source anyway, did they run the same request again with the question sharpened, did the draft survive to the thing that got sent. Not one of those is correctness. Together they bound it, and every one of them is a product metric with an owner.

But the move inherits more than the trick. Search's proxies bounded relevance because they ran on volume - millions of sessions a day, randomized traffic, a population that comes back - so noise averaged out and a real shift separated cleanly from a user who simply changed their mind. An ordinary enterprise deployment does not have that. At a few hundred sessions a day the same proxies cannot tell a worse model from a restless user, and a team that adopts them expecting search's resolution will read noise as signal for a quarter. What binds the move is traffic. Where you have the volume, the proxies work on a summarizer as well as they worked on a ranker; where you do not, they degrade to a sampled human read no matter how transactional the outcome.

There is a second place the move stops, past the volume line. Outcomes whose consequence is diffuse, delayed past any session, or borne by someone who never comes back - advice acted on three months later, a document read once by a person outside the product - give the proxies nothing to attach to at any volume. I do not have a measurement answer for that class. The honest response is a sampled human read plus a deterministic gate on anything irreversible, and that is a cost, not a metric.

Silent errors are only a problem while they are novel. The second occurrence of a failure class is a regression test and costs nothing. The first one is the real event, and no architecture prevents it. What you actually control is how long it lives. The distance between 13 hours and 60 days is not a detection technology difference. It is the difference between a team that treats the first strange report as a lead and a team that treats it as a ticket.

The recursion has to terminate in something that differs in kind from the thing under test. Reach for an LLM judge and it does not: you have installed a second unvalidated model in the detection path, failing quietly through position bias, verbosity bias and self-preference, as Saurav Bhattacharya argues [7]. Shreya Shankar and her co-authors close the other exit: "users need criteria to grade outputs, but grading outputs helps users define criteria," so the criteria cannot be fixed before the outputs are seen [8].

The product's own outcome is the only candidate I know of that differs in kind, because a refund either exists in the ledger or it does not.

Peter Hochschild and his colleagues at Google hit the same wall with cores that computed wrong answers and emitted no error - "detecting CEEs, conversely, naively seems to imply a factor of two of extra work" - and answered it by moving the check to where the right answer was cheap to recognise, through application-level redundant checking and core isolation [9]. That is not an instrumentation strategy. It is a design decision about where a wrong answer is allowed to be expensive.

What the team owns, not the vendor

The product owns the outcome. Not a quality bar, an outcome: the thing that is true in some system when this worked, and false when it did not. Everything downstream, including which evals are worth writing, is derived from that sentence, and it cannot be derived from anything else.

The harness owns the tells. The model is fluent; the scaffolding around it is not. Failed and malformed tool calls, silent retries, empty results narrated as answers, an assertion that disagrees with what the system recorded - these are cheap, deterministic, and none of them needs a second model to read it.

Then there is the part teams skip. Own the complexity you expect the model to resolve. Every ambiguity you left in the prompt because specifying it was tedious, every branch you handed over because enumerating it was work, becomes a failure mode with no owner and no test. That is not an observability budget. It is a complexity budget, and engineering is the one spending it. (The budget metaphor breaks in the way that matters: nobody sets this one's ceiling and nobody approves the withdrawals, which is why it is always overdrawn before anyone checks it.)

Silence in the system tends to be downstream of silence in the org: where nobody is rewarded for raising a gap they would then have to own, the gap stays unraised, and no stack fixes that.

Hamel Husain is closest to where I land, and his advice is the most useful in the genre: remove all friction from looking at your data, and never stop looking [6]. The one place I would push is that looking is only as good as knowing what you are looking for, and that comes from the product outcome rather than from the reading session.

Questions to run on your own system

What is this system for, in one sentence a product manager would sign: are we enabling something new, or doing something we already did?
If we replaced a human step, what did we measure about that step before? Which of those measures still exists, and who owns it now?
When the model is wrong, what does it touch? Name the table, the ticket, the ledger row, the downstream call.
If it touches nothing we can query, what do people do differently when it is wrong - and which of those is already logged?
What does the system already record that could disagree with what the assistant said? Who reconciles the two, and on what cadence?
How does a wrong answer reach us today, and what is the lag between it happening and us hearing about it? That number is your real detection latency.
What did we hand to the model because specifying it ourselves was tedious? That list is your unowned failure surface.
When this is wrong, what does it cost, and can it be undone? Which wrong answers have to be caught before the user sees them, and which can be caught afterwards because they are reversible?

If you cannot say where a wrong answer would surface, you have not finished defining the product. Instrument the place it lands, not the model that produced it.

Everything that team needed was already in the refund thread. There was a ledger. The ledger knew. The assistant had made an assertion about the world and the world had a record that contradicted it, and the gap between those two facts was queryable in one line of SQL. The dashboard was green because nothing in it had ever been pointed at the ledger.

No tracing vendor was going to find that. What was needed was a decision, made before shipping, that "the assistant says it is done" and "the system says it is done" are two different facts, and that someone owns the difference. The system will tell you it is wrong. It will just tell you somewhere you are not looking.

References

Wei Wu, "When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime," arXiv:2606.14589, 12 June 2026. https://arxiv.org/abs/2606.14589
Dexing Liu, "Silent Failure in LLM Agent Systems: The Entropy Principle and the Inevitable Disorder of Autonomous Agents," arXiv:2606.08162, June 2026. https://arxiv.org/pdf/2606.08162
John Hodge, "The state of the OpenTelemetry GenAI semantic conventions (July 2026)," 17 July 2026. https://john-hodge.com/blog/opentelemetry-genai-semantic-conventions/
Bibin Kuruvilla, "LLM Observability: What to Monitor When Nothing Is Broken," 4 August 2026. https://obsium.io/blog/llm-observability/
César Migueláñez, "Detecting AI agent failure modes in production: a framework for observability-driven diagnosis," 26 March 2026. https://latitude.so/blog/ai-agent-failure-detection-guide
Hamel Husain, "Your AI Product Needs Evals," 29 March 2024. https://hamel.dev/blog/posts/evals/
Saurav Bhattacharya, "Who Grades the Grader? Your LLM Judge Is an Unvalidated Model in Production," 27 June 2026. https://dev.to/saurav_bhattacharya/who-grades-the-grader-your-llm-judge-is-an-unvalidated-model-in-production-pfi
Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, Ian Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences," UIST 2024, arXiv:2404.12272, 18 April 2024. https://arxiv.org/abs/2404.12272
Peter Hochschild, Paul Turner, Jeffrey Mogul et al., "Cores that don't count," HotOS '21, May 2021. https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf

Views here are my own and do not represent those of any employer, past or present.