Microsoft Details Maia 200 AI Accelerator Architecture at Hot Chips 2026

Overview

Microsoft used a Hot Chips 2026 presentation to give its deepest public look yet at the architecture of Maia 200, the company’s second-generation custom AI accelerator for Azure. According to ServeTheHome, the talk was “the deepest Microsoft has gone into the chip’s architecture and the many design tweaks they have made to improve its performance over the original Maia 100.” Microsoft’s own Azure Infrastructure Blog and a companion paper posted to arXiv the same day laid out the chip’s design philosophy and performance figures in parallel with the conference session.

What We Know

• Maia 200 is Microsoft’s second-generation in-house accelerator, built on a 3nm process and aimed at inference workloads in Azure data centers, according to ServeTheHome. Microsoft’s own Azure Infrastructure Blog describes it as being “engineered from the ground up for efficient inference at cloud scale.”
• The chip packs 140 billion transistors on an 820mm² die, paired with six stacks of HBM3e memory, according to ServeTheHome. A companion paper on arXiv — co-authored by 17 researchers, including Hot Chips presenter Prashant Ranjan of Microsoft and ETH Zurich’s Torsten Hoefler — puts the chip’s throughput at 10,145 Tflop/s of FP4 compute and 5,072 Tflop/s of FP8 compute, within a 750-watt TDP and 7 TB/s of HBM bandwidth.
• The architecture is built around what Microsoft calls Software-defined Local Access, or SDLA. The Azure Infrastructure Blog explains that SDLA “gives software direct control over how data moves between high-bandwidth memory and localized, highly specialized SRAMs,” separating control from data movement rather than “relying on implicit cache behavior.” The arXiv paper frames this as part of “a new class of Software Defined Locally Accessed Dataflow Architectures,” arguing the approach “shifts the focus from today’s thread-centric to data-movement-centric architecture.”
• Each compute tile combines a tile tensor unit, a tile vector processor with SIMD engines, a tile control processor, a DMA block, and an L1 cache, according to ServeTheHome, which reported the chip achieves an effective peak of 1.65 PFLOPS on attention benchmarks and reaches nearly 1.3 TB/second on BF16 AllReduce collectives.
• Networking is entirely Ethernet-based rather than a proprietary interconnect. Microsoft’s blog describes a “two-tier scale-up network” using “a variant of the HammingMesh topology,” where four accelerators form a directly connected quad, 48 form a rack-scale domain, and the full switched network scales to 6,144 accelerators. Microsoft says it has contributed this networking direction to the Ultra Ethernet Consortium’s AI base transport profile.
• Microsoft reports — in its own blog post, without independent third-party verification — that Maia 200 delivers “over 40% higher token generation under the same rack power budget” than other accelerators in Azure’s fleet when running the MAI-Thinking-1 model, and claims “approximately 30% better performance per dollar than the latest-generation GPUs in Microsoft’s fleet.” The company says these results came from testing across 6,143 matrix-multiplication shapes representative of production inference workloads.
• MAI-Thinking-1, the reasoning model Microsoft cited in its Maia 200 benchmarks, is the flagship of the seven proprietary MAI models Microsoft previously reported launching in June, where Microsoft had already said it was co-designing the model with “Maia 200 silicon.”

What We Don’t Know

• Microsoft’s performance and efficiency comparisons — the 40% token-generation gain and 30% performance-per-dollar improvement — are self-reported figures from Microsoft’s own blog post rather than independently benchmarked results, and the identities of the “other leading accelerators” used as the comparison baseline were not disclosed.
• Pricing, availability timelines, and customer deployment details for Maia 200 were not covered in the Hot Chips presentation, the Azure Infrastructure Blog post, or the arXiv paper.
• ServeTheHome’s live-blog reported a rounded cluster figure of “128 racks with 6000 chips,” while Microsoft’s own blog gives a more precise maximum scale-up domain of 6,144 accelerators; the discrepancy appears to reflect rounding in the conference slide versus the paper’s precise topology math rather than a contradiction.

Analysis

The Hot Chips disclosure signals Microsoft treating custom silicon as one layer of a broader vertical stack rather than a standalone product. Both the Azure Infrastructure Blog and the arXiv paper repeatedly frame Maia 200’s efficiency gains as inseparable from co-designed software — the Microsoft Collective Communication Library, inference kernels tuned to the SDLA dataflow model, and models like MAI-Thinking-1 built with the chip’s characteristics in mind. That mirrors a pattern already visible in Microsoft’s own MAI model launch, where the company said it was already seeing efficiency gains from co-designing models with Maia 200 silicon months before this architectural detail became public.