凡是容易验证的事情，都会被AI做掉
最近读到 Jason Wei 的一篇博客，《Asymmetry of verification and verifier's law》（《验证的不对称性与验证者法则》原文在这）。
Jason是 OpenAI 的研究员，思维链那篇论文的作者。他这篇博客不长，但他这篇博客把我最近一直模模糊糊在想的一件事，给理清楚了。
他的思考从一个简单的现象开始：有些事，做起来很难，验证起来却很容易。
数独是最好的例子。
解一道数独要反复试错，检查一份答案只要扫一眼。
软件工程也一样，做一个 Instagram 量级的产品要一个团队干几年，但它好不好用，任何人点几下就知道了。做和验，难度完全不对称。
也有反过来的。
写一篇文章不难，把文章里每个事实都核实一遍，比写它花的时间还多。
提出一个新的科学假说很容易，验证它可能要好几年。这一头是验证比做更难。
此外，验证的这种不对称性是可以人为改变的。
竞赛数学题本来不好验证，但只要提前把答案备好，验证就变成了对答案。
写代码本来不好验证，但配上一套覆盖完整的测试用例，就变成了跑测试。
你在事前多做一点准备，就能把一件难验证的事改造成好验证的事。
Jason抛出了一个法则，验证者法则：
训练 AI 解决一个任务的难易程度，和这个任务可被验证的程度成正比。凡是解得出来、又容易验证的任务，AI 终将全部拿下。
什么叫容易验证？他给了五条：
有客观对错；
验证一次只要几秒；
能同时批量验证一大堆候选答案；
噪音低，验证结果和真实质量高度相关；
奖励是连续的，能给一堆答案排出个好坏来。
他回头看了一眼 AI 的历史，凡是满足这几条的基准测试，基本都被解决了。
这其中的道理并不复杂。
神经网络靠海量的梯度更新来学习，验证越快越准，每一步的信号就越强，能迭代的次数就越多。
快速迭代能碾压几乎一切。
Google 的 AlphaEvolve 就是这么工作的：提出一个候选方案，验证，再提出，再验证，用蛮力般的猜测加检查，把一个个可度量的目标推到极限。
而且这条路不止于数学和代码，催化剂配方、气动外形，任何能被测量的东西，理论上都在AI的射程范围之内。
Jason的判断是，未来的智能会呈现一种锯齿状的边缘。凡是可验证的领域，AI 会强得不成比例；不可验证的领域，进展会慢得多。
这篇文章不长，但是对我的启发是AI 让执行效率大幅提升后，剩下的问题是「什么值得做」。
验证者法则则提醒我们，接下来该问的还有「什么可以被验证」。
对于LLM/Agentic的从业者而言，这是个非常重要的角度，可验证的事会加速塌向 AI。
人的角色和责任会向两个方向移动：
一个方向是去定义验证，把模糊的目标改造成可度量的目标，这是杠杆最大的活；
另一个方向，是守着那些真正难验证的东西，比如品味、判断、价值感的判断，这些恰恰是AI无法胜任的部分。
原文很短，值得读一下。下文【转】
Asymmetry of verification and verifier’s rule
Jul 15
Written By Jason Wei
Asymmetry of verification is the idea that some tasks are much easier to verify than to solve. With reinforcement learning (RL) that finally works in a general sense, asymmetry of verification is becoming one of the most important ideas in AI.
Understanding asymmetry of verification through examples
Asymmetry of verification is everywhere, if you look for it. Some prime examples:
Sudoku and crossword puzzles take a lot of time to solve because you have to try many candidates against various constraints, but it is trivial to check if any given solution is correct.
Writing the code to operate a website like instagram takes a team of engineers many years, but verifying whether the website is working properly can be done quickly by any layperson.
Solving BrowseComp problems often requires browsing hundreds of websites, but verifying any given answer can often be done much more quickly because you can directly search if the answer meets the constraints.
Some tasks have near-symmetry of verification: they take a similar amount of time to verify as to write a solution. For example, verifying the answer to some math problems (e.g., adding two 900-digit numbers) often takes the same amount of work as solving the problem yourself. Another example is some data processing programs; following someone else’s code and verifying that it works takes just as long as writing the solution yourself.
Interestingly, there are also some tasks that can take way longer to verify than to propose a solution. For example, it might take longer to fact-check all the statements in an essay than to write that essay (cue Brandolini's law: “The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.”). Many scientific hypotheses are also harder to verify than to come up with. For example, it is easy to state a novel diet (“Eat only bison and broccoli”) but it would take years to verify whether the diet is beneficial for a general population.
Improving asymmetry of verification
One of the most important realizations about asymmetry of verification is that it is possible to actually improve the asymmetry by front-loading some research about the task. For example, for a competition math problem, it is trivial to check any proposed final answer if you have the answer key at hand. Another great example is some coding problems: while it’s tedious to read code and check its correctness, if you have test cases with ample coverage, you can quickly check any given solution; indeed, this is what Leetcode does. In some tasks, it is possible to improve verification but not enough to make it trivial. As an example, for a problem like “Name a Dutch soccer player”, it would help to have a list of the famous Dutch soccer players but verification would still require work in many cases.
Verifier’s rule
Why is asymmetry of verification important? If you consider the history of deep learning, we have seen that virtually anything that can be measured can be optimized. In RL terms, ability to verify solutions is equivalent to ability to create an RL environment. Hence, we have:
Verifier’s rule: The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI.
More specifically, the ability to train AI to solve a task is proportional to whether the task has the following properties:
Objective truth: everyone agrees what good solutions are
Fast to verify: any given solution can be verified in a few seconds
Scalable to verify: many solutions can be verified simultaneously
Low noise: verification is as tightly correlated to the solution quality as possible
Continuous reward: it’s easy to rank the goodness of many solutions for a single problem
It’s not hard to believe that verifier’s rule holds true: most benchmarks that have been proposed in AI are easy to verify and so far have been solved. Notice that virtually all popular benchmarks in the past ten years fit criteria #1-4; benchmarks that don’t meet criteria #1-4 would struggle to become popular. Note that although most benchmarks don’t fit criteria #5 (a solution is either strictly correct or not), you can compute a continuous reward by averaging the binary reward of many examples.
Why is verifiability so important? In my view, the most basic reason is that the amount of learning that occurs in neural networks is maximized when the above criteria are satisfied; you can take a lot of gradient steps where each step has a lot of signal. Speed of iteration is critical—it’s the reason that progress in the digital world has been so much faster than progress in the physical world.
AlphaEvolve
Perhaps the greatest public example of leveraging asymmetry of verification in the past few years is AlphaEvolve, developed by Google. In short, AlphaEvolve can be seen as a very clever instantiation of guess-and-check that allows for ruthless optimization of an objective, which has resulted in several mathematical and operational innovations.
A simple example of a problem optimized by AlphaEvolve is something like “Find the s mallest outer hexagon that fits 11 unit hexagons.” Notice that this problem fits all five desirable properties of verifier’s rule. Indeed, my belief is that any solvable problem that fits those five properties will be solved in the next few years.
One thing about the types of problems solved by AlphaEvolve is that it can be seen as “overfitting” to a single problem. In traditional machine learning, we already know the labels in the training set and the significant test was to measure generalization to unseen problems. However, in scientific innovation, we are in a totally different realm where we only care about solving a single problem (train=test!) because it’s an unsolved problem and potentially extremely valuable.
Relation to P = NP
One related but different concept is the open question in computer science of whether P = NP. While both P = NP and verifier’s rule discuss asymmetry of verification, verifier’s rule does not make any claims about the time it would take for AI to solve the problem. In fact, in many of the above cases, such competition math problems, it takes far more computation to solve the problem than to verify a solution given that we already have the answer key.
Moreover, verifier’s rule is even broader in scope than P = NP because verifier’s rule also applies to non-computational tasks. For example, verifier’s rule would claim that AI would eventually be able to optimize non-computational endeavors like finding the best catalyst to speed up a chemical reaction, or the best aerodynamic car design for the fastest quarter-mile time, provided that we design the systems to measure those endeavors quickly and at scale.
Implications
Once you’ve learned about it, you’ll notice that asymmetry of verification is everywhere. It’s exciting to consider a world where anything we can measure will be solved. We will likely have a jagged edge of intelligence, where AI is much Smarter at verifiable tasks because it’s so much easier to solve verifiable tasks. What an exciting future to consider.