Silent failures

Traditional software mostly fails loudly. A null pointer throws, a request times out, a test goes red. The bug announces itself, and you go fix it. AI systems fail differently, and that difference is the whole problem. They fail silently: they return a fluent, confident, plausible answer that happens to be wrong, and nothing anywhere lights up red.

A retrieval layer drops the one passage that mattered and still hands back five chunks. A model invents a citation that looks exactly like a real one. A classifier assigns the wrong category with 0.97 confidence. In every case the pipeline reports success. There is no stack trace, because nothing crashed. The output looks like the output you wanted. That is precisely what makes it dangerous, and it is much worse in a domain where someone is going to act on the answer.

Why this is the failure mode that matters

The cost of a loud failure is bounded: it stops the line, someone notices, you fix it. The cost of a silent failure compounds. It ships. It gets trusted. It gets built on. By the time anyone notices the answers were subtly off, the wrong outputs are downstream in a report, a ledger, or a decision, and you are doing forensics instead of debugging.

Worse, silent failures are invisible to the metrics people usually watch. Uptime is green. Latency is fine. Error rate is zero, because the system never raised an error. Every dashboard says the system is healthy while it is quietly wrong.

The discipline: make wrong answers detectable

You cannot prevent every wrong answer from a probabilistic system. What you can do is engineer so that wrong answers are visible rather than silent. A few of the patterns I lean on:

• Ground every claim, and show the grounding. In InferLens, every sentence of an answer carries a citation back to the exact source passage. A wrong answer becomes a checkable answer: the reader follows the citation and sees the gap. Provenance turns a silent failure into an obvious one.

• Let the system refuse. A model that is allowed to say “the document does not contain enough information” will do that instead of inventing something. A refusal is a loud failure, which is exactly what you want.

• Record which path produced the answer. Axiom logs, per transaction, whether the local model or the fallback handled it. When quality drifts, the record tells you where it drifted instead of leaving you guessing.

• Keep a ground-truth oracle. Exact search makes a great baseline for measuring what an approximate index is quietly costing you in recall. I wrote about that trade-off in Exact over approximate. The general move is the same: keep a slow, correct reference you can diff the fast path against.

• Write evals that catch quiet regressions. Not “did it run” but “is the answer still right.” An eval suite is the closest thing AI systems have to a red test.

The mindset

Building reliable AI is mostly the work of converting silent failures into loud ones: provenance, refusals, recorded decision paths, oracles, evals. None of it stops the model from being wrong sometimes. All of it makes sure that when the model is wrong, you find out before your users do.