GPUs Are Arrays of Tiny TPUs with Different Trade-offs
"At a very high-level point of view, the GPU has a lot of tiny, tiny TPUs tiled across the whole chip. You can sort of think of scaling this thing down into a really tiny unit with a smaller matrix unit, smaller vector unit. And that is sort of what an SM is."
About this episode
On this episode of the Dwarkesh Podcast, host Dwarkesh Patel was joined by Reiner Pope, CEO of AI chip startup Maddx, for a technical deep-dive into how AI accelerators work at the transistor level. Pope, whose company Patel has invested in, walked through the fundamental building blocks of chip design from logic gates up to full production chips, explaining why low-precision arithmetic provides quadratic rather than linear speedups and why data movement costs typically dwarf actual computation costs. A central revelation was that in traditional CPU and GPU architectures, roughly 87% of circuit area and energy goes to moving data between register files and logic units rather than performing calculations, which motivated the introduction of systolic arrays and tensor cores. Pope explained that NVIDIA recently revised their performance specifications to better reflect the quadratic scaling with bit precision, now advertising 3x speedup for FP4 versus FP8 in their B300 generation chips, though he noted this still understates the theoretical 4x advantage. The conversation covered why FPGAs provide deterministic latency for applications like high-frequency trading despite being 10x less efficient than ASICs, how clock cycles work and what determines them, and the fundamental architectural differences between GPUs and TPUs. Pope offered a novel framing that GPUs are essentially arrays of tiny TPUs, with the key trade-off being that GPUs enable higher data movement bandwidth between vector and matrix units through massive parallelism, while TPUs use fewer large units that better amortize register file costs. Toward the end, Pope disclosed that Maddx is developing split-table systolic arrays designed to capture advantages of both GPU and TPU architectures while avoiding their respective bottlenecks.
Key takeaways
- Pope revealed NVIDIA changed B300 specs to show FP4 runs 3x faster than FP8, acknowledging quadratic bit-width scaling though still understating the theoretical 4x advantage.
- In traditional CPU and GPU cores, 87% of circuit area and energy goes to data movement between registers and logic units, with only 13% doing actual computation.
- GPUs are architecturally arrays of tiny TPUs, with GPUs trading off large systolic arrays for higher bandwidth through massive parallelism across streaming multiprocessors.
- Pope explained FPGAs are 10x less efficient than ASICs because lookup tables require 32 gates to implement what takes 3 gates in an ASIC, but cost only $10,000 versus $30 million for first unit.
- Running chips at slower clock speeds reduces total energy proportionally but does not improve energy efficiency since power consumption comes from bit transitions, not idle time.
- Maddx is developing split-table systolic arrays designed to enable large systolic arrays that can function as small arrays, bridging GPU and TPU architectural advantages.
- Systolic arrays solve the data movement problem by storing weight matrices locally and reusing them across many input vectors, achieving linear rather than quadratic communication costs.