When Every Expert Is Right and the Whole System Is Wrong

A doctor studies a scan she can't quite read. The software has read it too, and it hands her a number: 82 percent confident. That number does enormous work. It tells her whether to trust the machine or order a second scan, whether to reassure the patient or to worry. We have built a whole layer of modern life on numbers like this one. A moderator decides whether to pull a post. A bank decides whether to freeze a card. The number is supposed to mean something. What if it's broken — every individual part working fine, the flaw hiding in how the parts add up?

Many AI systems work like a committee of specialists. A chairperson reads each case and decides how the experts weigh in. Here the design splits in two. The chairperson can hand the case to the single best-suited specialist and report that answer, or it can poll several specialists at once and blend their opinions into one verdict. The appeal of a committee is that you can check it the way you'd check anything complicated: audit each member. If every specialist is well-calibrated — meaning when one says "80 percent sure" it really is right about eight times in ten — then surely the blended verdict is calibrated too. Audit the parts; if they all pass, the whole passes.

A new paper shows that for the blending kind, this isn't true, while the selecting kind is safe. When the chairperson picks one expert, that expert's answer is the committee's answer; there's no gap for an error to hide in. But when you always average several experts, only the average ever reaches you. And here is the leak: many different mixtures of opinion can collapse to the very same blended number. From the outside, you can't tell which mixture produced a given number. That's where the ghost gets in. If the data shifts and the chairperson starts leaning on a different mixture than before, the final confidence can drift even though every individual expert stays exactly as accurate as it was.

Picture five weather forecasters, each flawless for their own region. Most of the year you average them evenly. Then an odd season arrives, and one gets consulted 80 percent of the time instead of 20. Not one of them got worse; every check still passes, but the blend has tilted and the audit can't see it.

The damage lands where it hurts: the hard cases. Tested on genuinely ambiguous images, the ones where people themselves disagree, a standard blended model's confidence was wildly off. The authors propose two fixes, both retraining the model to rehearse harder on the cases it finds most uncertain. The plainer one cut the error several-fold. On those hardest images, the stronger fix made the confidence roughly twentyfold more accurate. Neither made the model timid: accuracy on those hard images held steady, around 63 percent either way.

The unsettling part is scale. Every experiment used a committee of four. The systems behind the largest language and vision tools run with hundreds or thousands of experts, and whether the fix survives that size is unknown. We've gotten good at auditing the parts of an AI. This is a proof that the parts can each pass while the whole still fails — and at the scale that matters most, we don't yet have a way to check.