AI’s Path to Mastery: Where Scoring Systems Exist, AI Follows

Author: Robonaissance

Compiled by Deep潮 TechFlow

Shenchao Overview: Go, chess, protein folding—why have these fields, seemingly requiring human intuition and creativity, been successively conquered by AI? This article reveals an overlooked pattern: what determines whether AI can surpass humans is not the difficulty of the task, but whether success can be scored. Once a field establishes a scoring system, it sets a countdown to its own conquest.

Selected from the series "Whatever You Ask For," about the one task machines cannot do for us.

On March 10, 2016, in a hotel room in Seoul, a machine made a move that no one in the room could understand, while its opponent was not present.

Lee Sedol went out for a cigarette. He was 33 years old, held 18 world championship titles, and was widely regarded as the strongest Go player of his generation. Having lost the first game the previous day, his earlier confidence had diminished as he entered the second game, and his mindset became more cautious. While he was outside, AlphaGo made its 37th move, and DeepMind researcher Huang Shi-Jie quietly placed the stone on the board on behalf of the machine.

The stone has landed on the fifth line.

To someone who doesn’t understand Go, this makes no sense. To those who do, it’s nearly absurd. In the opening and middle game, moves played so far from the edge are considered inefficient—they secure no territory and offer no defense, effectively giving away points to the opponent. This is exactly the kind of move a teacher would correct in a beginner. On the commentary panel, top professional Michael Redmond, live-streaming the game, initially thought there was a display error. He picked up a stone, then put it back down.

Lee Sedol returned to his seat and stared at the board for a long time without moving. Estimates of the duration ranged from about 12 to 15 minutes. Fan Hui, the European champion defeated by AlphaGo five months earlier, was watching from inside the building. After observing for a long time, he concluded: "This is not a human move. I've never seen anyone play like this."

He is right, and his assessment is quantifiable. DeepMind’s system has an internal estimation mechanism that, through learning from a vast number of human games, estimates the probability that a human would play a particular move in any given position. For move 37, this estimated probability is approximately one in ten thousand.

The machine made its move, and this single play won the entire game.

A comforting interpretation of this story is that the machine worked incredibly hard to learn from human masters and eventually caught up to the best of them. This interpretation is wrong—and the number one in ten thousand precisely illustrates why. A system that learns and imitates human moves has a ceiling: the limits of human play. What happened in Seoul was that a system was no longer constrained by that ceiling, because its goal was not to earn the approval of human masters, but to win.

This distinction points in a more useful direction than admiration. If you want to know which human activities have already been surpassed in terms of pure capability, and which are next, the question isn’t how difficult the activity is, how much creativity it requires, or how much intuition is needed. The question is far more mundane than that.

The question is whether success can be scored.

Chess lost 19 years ago, in a completely different way.

In 1997, Deep Blue defeated Kasparov using an architecture that was essentially a massive human expression. Its evaluation function—the component that assesses a position and assigns it a numerical value—was assembled and fine-tuned with the help of grandmaster consultants. Human chess knowledge was painstakingly extracted from human players and encoded into the machine. What the machine added was search capability: the ability to look ahead more moves, faster, without fatigue, and without losing focus.

This was a true victory and should be counted as one. But pay attention to its nature. Deep Blue’s understanding of chess is human understanding. Its advantage lies in speed. If you ask it why it evaluates a position in a certain way, the answer ultimately traces back to the person who instructed it to do so.

Twenty years later, DeepMind released a system called AlphaZero, which had a different form.

AlphaZero was given only the rules of chess. It was not provided with an opening book—the curated catalog of studied first moves that every serious program relies on. It was not given endgame tables. It was not given a single human game. It played against itself, starting with random moves, and adjusted its strategy based on what led to wins.

Approximately four hours later, DeepMind estimated its Elo rating had surpassed that of Stockfish 8, the strongest traditional program at the time. About nine hours later, it played 100 games against Stockfish under time controls, winning 28, losing none, and drawing the remaining 72. The paper reported that within 24 hours, it achieved superhuman levels in chess, shogi, and Go.

These figures are often cited without the accompanying context, which is crucial. The self-play games were generated on five thousand first-generation Tensor Processing Units, with an additional sixty-four second-generation units training the network, all running in parallel. Four hours of wall-clock time equates to four hours of computation—a scale of computing power that no individual, or even a handful of institutions, could assemble. The achievement is not that chess is easy to learn. The achievement is that, when you have that much computational power, human knowledge is no longer what you need.

Garry Kasparov had more reason than anyone alive to take this result personally, yet he wrote a remarkably gracious commentary in Science. He observed that AlphaZero does not play the dull, cautious, draw-oriented chess that everyone assumed a perfect machine would play; instead, it prioritizes activity over material, sacrificing pieces for positions that appear risky to him. He noted that traditional programs reflect the priorities and biases of their human creators, while AlphaZero wrote itself. He concluded that its style therefore reflects something more authentic than programmer preferences, and observed that it wins despite examining far fewer positions per second than the best traditional engines in the world.

This final detail is worth remembering: the traditional engine searched far more possibilities per second, yet it lost. AlphaZero’s advantage wasn’t that it looked harder, but that it knew where to look—and this knowledge was assembled solely from millions of self-play games and a rule about who wins. Nothing more.

The outcomes of shogi games are stranger to those qualified to understand them. Shogi is a Japanese board game in which captured pieces return to the captor’s hand, making the game’s volatility greater than that of chess, and it also has centuries of accumulated theory on how to safeguard the king. Strong players who observed machine games reported that the openings violated established theory, with the king moving to the center of the board at moments when every book says it should retreat to the corner. These games were nearly incomprehensible to trained eyes, yet they won.

Place the two systems side by side—the pattern is unsettlingly clear. Deep Blue combines human knowledge with machine speed. AlphaZero combines machine speed with the definition of victory, deliberately removing human knowledge. The version without human knowledge is better.

All of this does not mean the 2016 machine was flawless—the same Seoul match contained evidence.

In the fourth game, trailing three sets to none and fighting for dignity, Lee Sedol inserted a single stone between two of AlphaGo’s groups in the center of the board. This move later became known as the “Divine Move.” AlphaGo’s own estimation of the probability that a human would play this move was also around one in ten thousand—a remarkably symmetrical coincidence. The system was unable to respond. Its assessment of its winning probability collapsed over the next few moves, its play deteriorated, and it lost the game.

So in March 2016, there was still a hole in the machine, and a human found it under global scrutiny and immense pressure. This is worth stating clearly, not quietly glossing over. Equally worth noting is what followed: such holes were steadily patched, and the system’s successors no longer lose such games—top engines haven’t lost a single game to a human in serious competition for a long time. The 2016 result was a snapshot of a transition, not a permanent balance of power.

From these boards to everything else, what is transferred is not victory itself, but the mechanism. Chess and Go were destined to be the first to fall, for embarrassingly simple reasons: in these games, success is defined by rules with complete precision. You either win or you don’t; scoring is free, instantaneous, and uncontested. A system can play forty million games against itself over a weekend and obtain forty million unambiguous verdicts.

This points to the next place to look—not those simple tasks, but the ones with their own scorecards.

Proteins are chains of amino acids that fold into complex three-dimensional shapes the moment they are synthesized. The shape determines what the protein does, and the sequence determines the shape. Deriving the latter from the former has been an open problem in biology since the early 1970s—and it has remained stubbornly unsolved: whether through first-principles physical simulations, statistical analyses of evolutionary relationships, or decades of accumulated structural intuition, none have succeeded.

In 1994, a team of researchers led by John Moult did something about this fact: everyone in the field was claiming progress, but no one could verify it.

They created an exam. Since then, every two years, the Critical Assessment of Structure Prediction (CASP) releases protein sequences based on recently experimentally determined structures—sometimes even before determination is complete—and makes them available to the world. Any team can submit predictions. No one has access to the answers, because in some cases, the experimental structures do not yet exist. When the experimental structures are finally determined, the predictions are evaluated and scored against them.

Scores are measured using a metric called the Global Distance Test, ranging from 0 to 100, which can be roughly understood as the percentage indicating how close the predicted final structure is to the actual structure. Moult noted that a score of around 90 is informally considered equivalent to structures determined in a laboratory. For much of the assessment's history, the best predictions hovered around 60 points.

At CASP14 in 2020, a participant registered as group 427 achieved a median score of 92.4 across all targets. On the most challenging category—targets with no useful structural homologs to rely on—the median score was 87.0. The average error was approximately 1.6 angstroms, roughly equivalent to the width of a single atom.

Moult, who has served as chair since the competition began, took the audience through the history of the competition before presenting the charts. The charts illustrated the entire story in a single image: two decades of lines crawling near 60, then one line rising to a level no other line had ever reached.

Group 427 is AlphaFold2. Moult announced that the problem has been solved for single protein chains.

Qualifiers should be there, in every honest account. A single chain is not the whole of protein science. How proteins assemble into complexes, how they move, and how they behave inside living cells—all of these remain open questions. A closed grand challenge is a specific, precisely stated challenge.

And it is this precision that is the point of telling this story here. What brought down structural forecasting wasn’t that it was originally simple—half a century of failures suggests the opposite. What brought it down was that, in 1994, the field built itself a scoreboard.

Viewing CASP as an engineering endeavor rather than a scientific one, it provides: a clear and unambiguous measure of success applicable to any prediction, computable in seconds; an uncheatable ground truth, since the answers are physically determined in someone else’s laboratory; a continuous stream of new problems generated on a fixed schedule; and a thirty-year history of scored attempts, providing a graded record of what has been better or worse in this field.

A field that has done all of this has inadvertently prepared itself for automated attacks. The scoreboard is the prerequisite; everything else is engineering and computation, both of which have become cheaper every year for a long time.

This promotes with unsettling ease. Wherever a field has reached a consensus benchmark, a target is published. Machine translation has benchmark scores. Speech recognition has benchmark scores. Image classification has a collection of one million annotated images and an annual competition—everyone who has seen that competition knows how the story ends. The pattern isn’t that hard things fall first or easy things fall first. The pattern is that what gets measured falls first.

This raises obvious questions about everything that hasn't been measured yet.

The last line of defense should be those things that cannot be scored.

Writing is a matter of standard examples. No program can take a paragraph and return a number saying how good it is. Two competent editors may disagree. The same editor may disagree with themselves on different days. The quality of language is entangled with context, audience, purpose, and taste—none of which can be reduced to a measurable value. If machines need a scoreboard and language has no scoreboard, then language is safe.

This argument is valid. What happened to it is the most important reason in this entire narrative.

There is no objective metric for good writing in this field. Such a metric still does not exist. Instead, it creates a score using the only available material: human judgment itself.

This approach has a longer history than most people realize. In 2008, Knox and Stone described a system called TAMER, in which humans observed agent behavior and provided evaluations, which were then used to train a model to predict what humans would say. The model, rather than humans, subsequently provided training signals. In 2017, Christiano and colleagues published a work widely regarded as the direct origin of current practices, applying this idea to agents playing Atari games. In 2019, Ziegler and colleagues applied it to language models, and most of the terminology used today was established in that paper. In 2022, Ouyang and colleagues demonstrated this approach at an industrial scale on instruction-following tasks, and shortly thereafter, most people on Earth were exposed to these成果 without realizing it.

Its core mechanism is worth understanding in detail, as it is much simpler than it sounds.

A person sits before two passages, both generated by a model from the same prompt. The task is not to score them—scoring is something humans do poorly, since one person’s seven is another’s five, and even the same person is inconsistent within a single afternoon. The task is simply to say which one is better. This is a judgment humans can make reliably, and the mathematical methods for converting a large set of such comparisons into a consistent scale are older than the field itself, dating back to Bradley and Terry’s 1952 work on paired comparisons.

Consider the working conditions under which these judgments are made, as they ultimately have an impact. This person makes hundreds of such judgments per hour, paid to compare responses against a written guideline that explains what customers consider to be better answers. Some pairs are nearly identical. Others involve topics this person knows nothing about—in such cases, the answer that appears more confident and better organized is often selected, regardless of whether it is more accurate. This is not a criticism of these individuals. It is a description of what anyone would likely do under these conditions, and the resulting selections become the raw material.

Collect enough of these choices, and you can train a second model whose task is to predict which of two texts humans will prefer. This second model outputs a number—and a number is exactly what has been missing all along.

Take a close look at what is being established here, as it’s easy to overlook. The optimized score is not about facts of the world—it’s about our model. It’s a compressed, learned imitation of judgments made by a specific group of people hired at a particular time, working under specific instructions, and as with everyone else, growing tired in the afternoon. CASP scores predictions based on physical reality determined in laboratories, while this system scores text based on human-validated statistical profiles.

This portrait is useful. But it is necessarily an approximation, with differences from the object it depicts that no one has fully understood. Anything about our actual preferences that our preference model fails to capture is not in the target, and therefore will not be optimized—and thus remains vulnerable to any influence—while a powerful optimizer has no reason to protect this quantity.

This is a matter of construction; no claims are made here about how severe the consequences might be. The current point is narrower and harder to refute. The categories of things that cannot be scored are smaller than they appear. If a field resists measurement, a measure can be constructed from human preferences and progress can continue. This defense only holds when no one has thought to create a score.

This leaves a practical question and a viable answer.

Choose any activity you like: a job, a craft, a profession, a task you've spent years learning to do well. Ask three questions.

First, can success and failure be reliably distinguished? Not perfectly, and not by everyone, but consistently enough that competent evaluators generally agree. Games pass this test easily. Protein structure prediction passes because laboratories eventually produce definitive answers. Writing does not pass in an absolute sense, but it does pass in a relative sense, as people typically can identify which of two attempts is better.

Second, can this judgment be generated at low cost and at scale? This question determines timing, not possibility. A free and instant judgment—such as one made in a game—means the system can autonomously generate millions of training examples. A judgment requiring a laboratory is slower but still feasible, with decades of archived ratings to draw from. A judgment requiring paid human reviewers to read text is expensive, which is why so much effort has been invested in training models to mimic these humans and remove them from the loop.

Third, the question is not when one domain will fall, but what happens afterward: Is this score truly what you want? The score in a game is what you want, because in a game, winning is by definition the entire point. The experimentally measured protein structures are very close to what you want. A learning model based on annotator preferences is not what you want—it is a portrait of what you want, and there is a gap between a portrait and a face.

Test your own work with these three questions. Most people find the first answer comes quickly and is unsettling. Most of what we call skill, at least in relative terms, can be scored when capable individuals are placed side by side and given two attempts. The second question is where true uncertainty lies, because costs and scale are moving targets—and they’ve been moving in one direction for a long time. The third question is rarely asked by anyone, yet it determines what your field looks like on the other side.

It’s worth practicing this slowly on ordinary matters. Take radiology: Can success and failure be distinguished? Yes, and very precisely, because diagnoses are ultimately confirmed or refuted by patient outcomes. Can judgments be produced cheaply and at scale? Essentially yes, because hospitals have been accumulating scored samples for decades in the form of images and confirmed results. Is this score what you actually want? Here the answer becomes complex, because the score measures consistency with recorded diagnoses, while what you truly want is good patient outcomes—and these two things align most of the time, but diverge precisely in the most critical cases.

Now take something that seems safer—management, for example. The first problem is already difficult because experts disagree on whether a given manager is effective, and these disagreements don’t resolve quickly. The second is even harder, because the outcomes of management decisions take years to materialize and are entangled with everything else that happens. The third is the hardest, because any alternative metric proposed for good management—whether retention rates, engagement scores, or output per person—is clearly not the thing itself. Based on this diagnosis, management isn’t safe because it’s profound. It’s safe because it’s unmeasured—and that’s a different, less appealing kind of protection.

Machines have already claimed the optimization axis. With sufficient computation, finding good moves on any clearly defined objective is no longer a competition—and the results are often not just stronger than ours, but also more alien, reaching places our traditional methods taught us not to look. This is not prediction. This is a description of chess, Go, shogi, protein structures, and an increasing number of things that once sat on the list of tasks only humans could do.

What remains is the target itself. Someone decided what the score would be. In each of the cases above, that decision took an afternoon, and every extraordinary outcome emerged from it, remained faithful to it, and was indifferent to anything it overlooked.

So the last question is worth reflecting on: In your work, are there already scores? If so, who wrote them, and did they think of you when they did?

This is Part Two of the "Whatever You Ask For" series, about what the machine ultimately needs from us—and how poorly we’ve done.