AI Lab Automated Scientist Can Optimize Hyperparameters But Cannot Do Lateral Thinking

What was said

"Current closed models that we can access today, they don't seem to be that great at selecting what the next experiment should be in a given track, and they don't seem to be able to step back and do the lateral thinking of, wait a minute, this track doesn't really make sense. Let's go back to first principles. Often I had to catch infra bugs myself by prompting the right question to Claude."

Zhang used Claude 4.6 and 4.7 extensively for AI research automation and found models excel at hyperparameter optimization and executing specific experiments but fail at higher-level research strategy. The AI cannot determine when to abandon unproductive research directions or step back to reconsider fundamental assumptions. Zhang suggests this represents a key bottleneck in fully automated AI research, though he notes upcoming models like Mythos may address these limitations.