Helpful Is Not Something a Machine Works Out for Itself
A stripped-down AI beat a model more than a hundred times its size, for one reason: people sat down and told it which answers were better. What it learned was their taste.
Picture something that has read nearly everything ever written and still cannot take an instruction. You ask for a short summary and get a meandering essay. You ask a question and get a list of related questions. The sentences are fine. They're just not answers to you.
That failure comes straight from how these systems learn. A model like GPT-3 is trained on one exercise, repeated across an enormous library: given a stretch of text, guess the word that comes next. Do that long enough and you get a machine that can write in any voice on any subject. But guessing what comes next and doing what a person asks are separate skills, and they come apart precisely where you'd want them to hold. The working assumption in the field was that size would close the gap on its own. Build the thing big enough and obedience to intent would arrive as a side effect. GPT-3 had 175 billion internal dials, tuned by all that reading.
A team at OpenAI set up a contest that should have been a rout. They took the same system shrunk to 1.3 billion dials, under one percent of the size, and changed the training rather than the scale. People wrote good answers to real prompts by hand. Then they ranked the model's own attempts, best to worst, over and over, and the model was tuned to chase whatever those people had preferred. The difference between a student who has only read textbooks and one whose homework comes back marked.
Shown the two side by side without labels, people preferred the small, coached model. A machine with more than a hundred times fewer dials gave better answers than the giant, because someone had bothered to show it what a good answer looked like.
Coaching the full-size model widened the gap. Judges chose its answers over the untouched original about 85 percent of the time. It fabricated roughly half as often: 21 percent of the time on tasks with a checkable answer, against 41 percent. On a test of truthfulness it gave replies that were both true and useful about twice as often. Asked to be respectful, it produced around a quarter less toxic language. And the coaching was cheap. All that extra training consumed under two percent of the computing power that had gone into building GPT-3 in the first place.
Preferred by whom, though. The rankings came from a pool of about forty hired contractors, mostly English-speaking. The model did not learn human values. It learned that group's taste. It still invents facts, still turns toxic, and will still carry out a harmful instruction from anyone who phrases it politely enough. What the experiment demonstrated is that these systems aim with real precision at whatever target you hand them. Size supplies the capability; the target supplies the direction. Every assistant trained this way since points at a target some particular group of people chose. Helpful is not a property the machine discovers. It's a setting, and somebody sets it.