Better Instructions Can Break a Weaker AI

Hand a new hire a thorough onboarding doc and they find their feet faster. The intuition is so plain it barely feels like a claim: better directions, better work. So when teams began dropping small README-style files into their codebases — short notes explaining how a project is laid out, meant to be read not by people but by the AI assistants that now write a lot of software — few fussed over the wording. The files, often named something like AGENTS.md, were treated as a nice touch. A convenience, not a lever.

A group of researchers decided to take the wording seriously. What if you refined one of these files the way a good teacher refines a lesson, by watching exactly where the learner trips? They built a loop, and the elegant part is that one AI played every role. It invented a batch of fake bugs in a codebase. It tried to fix each one using the current instructions. It graded its own attempts. Then it rewrote the instructions to patch whatever had just gone wrong, and ran the whole thing again. No bigger model, no new tools, no extra training. Only the directions got smarter.

The payoff was real. Set loose on a standard set of real bug-fix tasks pulled from active open-source projects — the kind of issues that pile up in any large codebase — the assistant with refined instructions solved 33 percent of them, up from 25.5 percent with no guidance at all, and a few points above a generic auto-generated file. Across hundreds of tasks, that gap is dozens of bugs that now get fixed instead of abandoned. But look closer at what changed. The guidance didn't raise the quality of each attempt; that stayed flat, the same share of patches landing whether the instructions were good or absent. What it changed was the attempt rate. The assistant tried on 56 percent of tasks with refined guidance versus 42 percent on its own. More swings at the same batting average means a lot more contact.

Then the story turns. The researchers fed those same polished instructions to a smaller, less capable model — and its performance dropped, from 28.4 percent on its own to 27 percent with the very guidance that had lifted the stronger one. The help made the weaker AI worse. Picture handing a nervous first-week intern a manual written for a senior engineer: more precision, more context, more to hold in mind, all of it pitched to a reader who can carry the load. The smaller model was simply overloaded by what the larger one had needed. Whether instructions are good, it turns out, depends on the mind reading them as much as on the words themselves.

The case has cracks. It ran on a single benchmark. And in the refining, the instructions grew about 63 percent longer, so the researchers can't yet say whether the gain came from wiser instructions or simply longer ones — the two aren't the same thing, and the paper can't separate them. But the lesson holds, and it's a strange one. The cheapest way to improve these systems may be to write to them more carefully. The catch is that you can't write the note once. Every model you deploy may need its own — its own onboarding, tuned to how it happens to think.