Value Functions Can Help AI Assess Its Direction Early, but Whether the Goal Is Right Still Requires Separate Verification

Turn Biological Analogies into Testable Questions

The WeChat public account proposes looking to biological mechanisms of learning and cooperation for ideas to improve AI capabilities and achieve alignment. One question worth pursuing is whether an agent can judge whether its current action is worth continuing before the final outcome arrives. In the original interview, Ilya Sutskever explained the potential role of value functions through self-assessment while learning to drive. My judgment is that this approach has research value because it points to specific mechanisms governing how and when feedback arises. But it is not enough to treat learning efficiency, generalization, and goal selection as a single problem that has already been solved.

Evidence: [5] [3]

What the Dopamine Experiments Support

The 1997 paper by Schultz, Dayan, and Montague documented changes during reward learning in monkeys: unexpected juice triggered a brief response in some dopamine neurons; once the monkeys learned the cue, the response shifted to the cue's appearance; when expected juice failed to arrive, activity fell at the moment the reward was due. The authors explained these observations in terms of reward prediction error. The most useful connection here is that a learning signal can arise from the difference between an outcome and an expectation. The limits of the evidence must remain clear: this was neuronal activity recorded in specific experiments. It cannot explain all dopamine functions, nor does it justify directly calling dopamine the Critic responsible for value estimation.

Evidence: [1]

What Intermediate Value Estimates Do—and What They Require

The value function in the paper estimates the expected discounted sum of future rewards, while temporal-difference error links the current reward to value estimates at adjacent time steps. This suggests that an agent may use changes in value estimates to update its judgment even before completing a task. The appeal of this mechanism is that it reduces the wait imposed by relying entirely on feedback at the end, but it depends on estimates that effectively reflect subsequent returns. Ilya's comments in the interview also retain clear limits: value functions can make reinforcement learning more efficient, while the relevant work might still be accomplished more slowly without them. His more fundamental concern remains that models generalize less well than humans. An efficiency benefit therefore cannot directly serve as evidence that the generalization problem has been solved.

Evidence: [1] [3]

Learning from Experience Remains a Research Agenda

In Welcome to the Era of Experience, Silver and Sutton advocate letting agents learn from ongoing experience generated through interaction with their environments, and discuss value functions, exploration, world models, and long-term decision-making. This places value functions within a set of interconnected research questions; it does not establish that value functions alone are sufficient for continual learning. The authors also point out that extended autonomous action reduces opportunities for human intervention and increases demands concerning trust, responsibility, and explanation. This suggests that evaluating systems that learn from experience requires examining both whether they improve their behavior by learning from consequences and whether humans can understand and intervene in time. Longer periods of autonomous action do not, by themselves, prove greater reliability.

Evidence: [2]

Long-Term Returns and Moral Values Must Be Kept Separate

A value function estimates future returns under a specified reward; whether that reward objective is worth pursuing still requires independent judgment. Estimating returns more accurately does not logically guarantee that the objective better reflects human intentions. In its January 22, 2026 explanation of its new constitution, Anthropic said its training design added explanations of the intentions and reasons behind behaviors, with the aim of helping the model apply principles in new situations while retaining hard constraints on some high-risk behaviors. The company also acknowledged that actual outputs may fall short of the constitution's ideals. This supports a bounded judgment: explaining the reasons behind principles is a change in training design whose effectiveness still needs to be demonstrated through behavior. It cannot directly be taken as evidence that the model already has reliable moral judgment.

Evidence: [1] [4]

What Evidence Would Support a Stronger Conclusion

For now, the more defensible conclusion is that biological reward learning offers substantive computational inspiration, and that value functions merit further research as a mechanism for improving learning efficiency. To advance this judgment to a claim of a breakthrough in continual learning would require publicly available algorithms and reproducible comparisons showing whether efficiency gains carry over to new tasks and environments. To advance it to safe alignment would also require evidence of how the system acts when goals conflict or principles are difficult to apply. These are the verification requirements proposed in this article. Ilya's research intuition, Silver and Sutton's research outlook, and Anthropic's explanation of its training design each offer direction. Together, they do not yet constitute proof that these capabilities have been achieved.

Evidence: [3] [2] [4]

What to watch next

• Whether public, reproducible value-function experiments emerge that can distinguish improvements in learning efficiency from improvements in generalization across tasks.
• Whether the benefits of continual learning through interaction with an environment persist in new environments while preserving effective opportunities for human intervention.
• Whether explaining the reasons behind principles produces reproducible behavioral improvements in unfamiliar situations and when goals conflict.