MLPerf Training v6.1: NVIDIA Scales to 10,240 GPUs While Google Keeps Pace
The MLCommons has released the next round of independent AI benchmarks with MLPerf Training v6.1, introducing new workloads like Qwen3.5-397B GRPO and DLRMv4-HSTU. This iteration is particularly significant as it features the latest high-end accelerators, including NVIDIA's Vera Rubin and Google's TPU7x (Ironwood), alongside AMD's Instinct MI350X/355X series, providing a direct comparison of the industry's leading AI training hardware.
As AI training systems grow in complexity, models require more parameters, massive datasets, and unprecedented computational power. These models often exceed the memory capacity of a single accelerator or even an entire server, necessitating distribution across hundreds or thousands of GPUs. The goal is not just to increase the number of accelerators, but to ensure they work efficiently through high-speed interconnects, memory, and optimized software. Scaling is the key to reducing training times from months to mere days or weeks.
The scale of these clusters is staggering. NVIDIA has demonstrated massive scale with its own supercomputer comprising 10,240 GB300 accelerators. Google has scaled its training to a cluster of 8,192 TPU7x units. In practical industrial applications, the scale goes even higher; for instance, SpaceX's Colossus-2 system reportedly utilizes 110,000 GB200 and 440,000 GB300 accelerators working in unison.
The Role of Software Optimization
While hardware scale is vital, software efficiency has become a decisive factor in training performance. With the rise of AI coding agents, even kernels and training processes are being monitored and fine-tuned. This software-driven optimization played a significant role in the MLPerf Training v6.1 results. AMD reported performance gains of approximately 20% through software tuning on identical hardware, while NVIDIA saw improvements ranging from 20% to 85%, depending on the specific model and workload.
Scaling at the (Preliminary) Limit
Comparing the largest clusters is challenging because not all configurations submitted results for every benchmark. However, the disparity in maximum scale is evident: NVIDIA's submissions reached 10,240 GB300 and 1,024 Vera Rubin units, Google reached 8,192 TPU7x, and AMD's largest submission was 128 Instinct MI355X accelerators. This highlights a current limitation for the Instinct generation, which is expected to see significant scale-out improvements with the upcoming Instinct-400 family, Helios racks, and the Infinity Fabric/UALink interconnects.
Maximum Accelerator Scaling Comparison
- NVIDIA GB300 (Theia-jhb, 160x NVL72): 10,240 accelerators (Reference)
- Google TPU7x (Ironwood): 8,192 accelerators (80% of GB300 performance)
- NVIDIA Rubin (Hecate, 16x NVL72): 1,024 accelerators (10% of GB300 performance)
- AMD Instinct MI355X (Crusoe, 16x System): 128 accelerators (1.25% of GB300 performance)
- AMD Instinct MI350X: 32 accelerators (0.31% of GB300 performance)
- AMD Instinct MI300X (Krai, 1x System): 8 accelerators (0.08% of GB300 performance)