AI Researcher Reveals Why LLM Reinforcement Learning Is Fundamentally Less Efficient Than AlphaGo

What was said

"In an untrained model, if your policy has no chance of sampling blue, then you will never get a signal. You spend most of training in this low pass rate regime and you're getting very little signal. Once you're at zero percent, it's not at all obvious how you get to a non-zero pass rate."

Zhang explained why policy gradient reinforcement learning used in LLMs is inherently inefficient compared to AlphaGo's Monte Carlo Tree Search approach. In early training with a 100,000-token vocabulary, random exploration yields almost no learning signal as the model must stumble upon correct answers by chance. AlphaGo avoids this trap by using MCTS to provide improved action labels at every state, maintaining a stable supervised learning signal throughout training rather than depending on rare successes.