Former Google Researcher Says Architectural Choices Like Transformers No Longer Matter for Go

What was said

"Architecture choices don't matter that much. Transformer versus ResNet, we're at the speed of GPU where the size of the model is not so big that this really matters. You can actually simplify this setup quite a lot. Some of the auxiliary supervision objectives that Katago developed aren't really necessary if you have a strong initialization."

Zhang found that with modern hardware and proper initialization against existing strong models, many architectural innovations and training tricks developed for Go AI systems are now obsolete. His experiments showed ResNets and Transformers perform comparably, and complex distributed training infrastructure can be replaced with simpler synchronous approaches. This validates aspects of the 'bitter lesson' that raw compute and scale matter more than algorithmic sophistication, though Zhang notes initialization strategy remains critical.