MLPerf Inference v6.1: A New Era of AI Benchmarking
MLCommons has officially released the results for MLPerf Inference v6.1, marking a significant milestone in the evaluation of artificial intelligence hardware. This release introduces a record-scale 512-accelerator submission, new benchmarks for retrieval-augmented generation (RAG), and the first peer-reviewed performance data for NVIDIA’s Vera Rubin platform. The September 16, 2026, update includes submissions from 30 organizations, covering 120 distinct systems and 486 data-center and edge results. This comprehensive round spans a wide spectrum of hardware, from individual edge devices and workstation GPUs to massive rack-scale systems equipped with hundreds of accelerators.
The results highlight a shifting landscape in AI inference. AMD demonstrated large-scale scaling with its MI355X accelerators, NVIDIA’s Blackwell platform maintained strong competitiveness in conventional deployment sizes, and Vera Rubin made its debut in the benchmark. Intel also expanded its participation with the Arc Pro B70 and additional Xeon 6 configurations. Rather than identifying a single dominant platform, MLPerf Inference v6.1 illustrates how system scale, workload characteristics, software optimization, and accelerator architecture interact to determine overall inference performance.
AMD’s Record-Breaking 512-GPU Submission
The most prominent result in this round is AMD’s 512-GPU Instinct MI355X submission. MLCommons identified this as the largest system ever submitted to MLPerf Inference, surpassing the previous record of 288 accelerators. The system, submitted by Crusoe, consists of 64 nodes using AMD Instinct MI355X accelerators. In the GPT-OSS-120B benchmark, the cluster achieved approximately 5.75 million tokens per second in Offline mode and 5.39 million tokens per second in Server mode. For the DeepSeek-R1 model, it reached approximately 2.90 million tokens per second in Offline mode and 2.40 million tokens per second in Server mode.
These figures represent aggregate cluster throughput rather than individual GPU performance. The significance lies in scale-out inference, where modern large-model serving requires multiple accelerators and nodes. Consequently, interconnects, networking, distributed scheduling, and software orchestration become critical components of total system performance. AMD reported a 95% scaling efficiency for GPT-OSS-120B at a 72-GPU scale, while NVIDIA’s 288-GPU GB300 submission achieved a 99% scaling efficiency. These metrics underscore that raw accelerator throughput is insufficient for hyperscale inference; high scaling efficiency indicates that the software and infrastructure stack are effectively converting additional accelerator capacity into useful throughput.
NVIDIA Blackwell and Vera Rubin Performance
While AMD captured attention at extreme scale, NVIDIA’s Blackwell platform remained highly competitive in conventional 8-GPU and 72-GPU configurations. In the Llama 2 70B benchmark, NVIDIA’s GB300 configurations led competing systems at both scales. Additionally, the Llama 3.1 8B results showed a performance advantage for the 8-GPU GB300 configuration compared to an equivalently sized MI355X setup, with NVIDIA reporting an advantage of up to 17%.
The Debut of Vera Rubin
MLPerf Inference v6.1 also marks the first peer-reviewed benchmark appearance of NVIDIA Vera Rubin. The Vera Rubin NVL72 platform participated in the DeepSeek-R1 benchmark with 72-GPU and 36-GPU configurations. According to NVIDIA’s submission highlights, the Vera Rubin NVL72 delivered up to 2.5x the DeepSeek-R1 throughput of the prior generation and up to 3.7x the throughput on Qwen3-VL. These results provide an early, independently benchmarked reference point for the next-generation architecture, though Vera Rubin remains a preview platform in this round.
MLPerf has therefore started to shift from asking “How fast can this accelerator run a model?” toward more system-oriented questions such as “How efficiently can this platform execute an end-to-end AI workload?”
New Benchmarks for Agentic AI and RAG
This release expands beyond traditional single-model inference by introducing two new benchmark areas: End-to-End RAG for data-center deployments and Agentic Edge Inference for single-user edge devices. The RAG benchmark evaluates a multi-stage pipeline including embedding, retrieval, re-ranking, and LLM-based answer generation. This approach is closer to how enterprise RAG systems are actually constructed, where performance depends on interactions among multiple models and infrastructure components. Similarly, the new Edge Agentic Inference benchmark reflects the move from single-turn model execution toward multi-step workloads such as agentic coding.
Intel’s Expansion and the Role of Software Optimization
Intel expanded its presence with the Arc Pro B70, which participated in workloads including Llama 3.1 8B, Llama 2 70B, GPT-OSS-120B, Whisper, and End-to-End RAG. Intel reported that on the same four-GPU hardware used in v6.0, GPT-OSS-120B performance increased by 36% in Server mode and 27% in Offline mode, attributing the improvement primarily to software-stack maturation. Furthermore, Intel’s Xeon 6 participation grew from two SKUs to five, with CPU inference results increasing from 24 to 35. Intel reported a 2.4x throughput increase in the Llama 3.1 8B Server scenario on the Xeon 6980P compared to v6.0.
A clear theme in MLPerf Inference v6.1 is that performance improvements are increasingly driven by software as well as new silicon. Many of the strongest gains came from new hardware and software/algorithmic optimization. AMD highlighted additional performance from the same MI355X hardware after another cycle of ROCm development, while NVIDIA reported that some v6.1 results were as much as 1.6x higher than v6.0. This trend implies that a processor’s practical inference performance is determined by the complete stack, including hardware architecture, memory subsystem, interconnect, compiler, kernel implementations, runtime, and model optimization.
Conclusion: A Systems-Engineering Perspective
MLPerf Inference v6.1 does not establish a single universal performance leader. Instead, it shows different strengths at different deployment scales and workload types. AMD demonstrated the ability to scale to a record 512 accelerators, NVIDIA’s Blackwell systems remained competitive in mainstream configurations, and Vera Rubin provided the first peer-reviewed results for NVIDIA’s next-generation platform. For AI infrastructure teams, the most useful approach is to evaluate these results against the specific model, latency target, deployment scale, hardware configuration, and software stack relevant to their intended production workload. The benchmark confirms that AI inference is becoming a complex systems-engineering problem where software maturity is as critical as hardware innovation.