Everyday Apparatus
IntelligencearXiv3 min read1 month ago

The AI That Reasons in Private Can Be Taught to Show Its Work

A new breed of AI seemed to think in an unreadable private language. It turns out a whisper of plain words at each step is enough to make it watchable.

A read of How Transparent is DiffusionGemma? · arXiv:2606.20560

Guided read0:00 / 4:25

Autoregressive model

Generates output one token at a time, left to right; each step is word-level and discrete, making the reasoning sequence inherently readable.

Denoising steps

The iterative passes a diffusion model makes from a noisy starting state toward a coherent output; what flows between these steps is what this paper studies.

Self-conditioning

The continuous vector a diffusion model passes to itself between denoising steps, carrying intermediate computational state — the thing the paper's bottleneck is applied to.

Opaque serial depth

The paper's metric for how much of a model's computation runs through stretches no human can interpret; higher means less transparent.

Token bottleneck

The intervention: squeezing the continuous latent vector through the model's vocabulary layer at each step to produce a small set of real words instead of raw activations.

What it’s not claiming · The paper does not claim that DiffusionGemma is now completely transparent or that the eight‑token bottleneck provides a full, causal explanation of its decisions for every diffusion model.

The worry about artificial intelligence that anyone can put into words is the simplest one: what if the machine is working something out that we can't see? Not a robot uprising. Just the ordinary unease that if you can't follow how something reasons, you can't catch it the moment it goes wrong. For a long time that fear stayed abstract. Lately it has taken on a shape.

Most of today's AI thinks out loud. Ask it a hard question and it writes its way to an answer, one sentence after another, and you can read the trail it leaves. But a newer kind of system doesn't work like that. It doesn't write a thought and then the next thought. It starts with a rough draft of an answer and revises the whole thing, over and over, the way a blurry photo sharpens as you wait. The trouble is that the revising happens in a private mathematical space, a kind of internal muttering no one had figured out how to translate.

Researchers found a way to measure exactly how private. Their question was concrete: how many steps does the machine take in the dark before anything surfaces that a person could actually check? For the think-out-loud kind of AI, that number is already large. For the revising kind, it was nearly thirty times larger. Picture following someone's reasoning through a glass wall. With the familiar system, the glass fogs over now and then. With the new one it is all but opaque. Twenty-eight times harder to see through.

Then came a move so plain it sounds like cheating. What if you forced the machine to leave itself a note at every step? Not an essay. Eight words, only the ones it felt surest about, the handful that cleared a low bar of confidence. Between each pass of revision, all that internal muttering now had to squeeze through an opening just wide enough for a few readable words. A margin note before turning the page.

The fog lifted. The gap between this system and the readable kind fell from nearly thirty times worse to near zero, close enough to call them even. And here is the part that should stop you: the machine did not get worse at its job. On the punishing tests — competition math, writing code, graduate-level science — it scored the same as before. You could now read its private deliberations, and it still solved the same problems at the same rate. The opacity, it turned out, was never built into the thing. It was a choice nobody had made yet.

Here is the thrilling part, and also the unfinished one. Readable is not the same as truthful. The notes look like plain English now, but nothing guarantees the machine isn't tucking real meaning into the spaces between the words, hiding in plain sight. The researchers said as much themselves. They have shown that the surface can be made legible. Whether what lies beneath can be trusted is a different question, and the door they unlocked is still open.

Where this sits

Open question

How much of DiffusionGemma’s underlying computation remains hidden after reducing each denoising step to just eight tokens, and can we be confident those tokens capture the model’s full reasoning across all possible inputs?

Next readThe Cancer Gene You Knew Was Doing a Second Job You Didn't