Architecture search reveals transformers need less structure at trillion-parameter scale

What was said

"Scaling laws define architecture. When we are talking about Transformers being this revolutionary thing, we are talking about maximum scale. The reason why transformer architecture and attention mechanism is such a brilliant architecture is the fact that it is unstructured. There is no structure. The larger neural networks that you make into infinite size, the larger neural networks you make, the more you want them to become less and less structured. As soon as you start adding a little bit of bias in that architecture at scale, things become completely messed up."

Liquid AI's automated architecture search across models from 10 million to 72 billion parameters revealed a fundamental scaling principle: larger models require less architectural bias and structure, while smaller specialized models benefit from complex gating and nonlinearity. This insight challenges assumptions about optimal architectures and suggests transformers' dominance at frontier scale is mathematically justified.