d-Matrix Raptor hits 100 TB/s with stacked 3D-DRAM for AI inference
d-Matrix has unveiled a new accelerator architecture called Raptor at Hot Chips 2026, aiming to solve memory bottlenecks in generative AI inference by stacking logic directly on top of DRAM. Unlike traditional High Bandwidth Memory (HBM) stacks placed beside the compute chip, Raptor uses a vertical 3D-DRAM design where the logic die sits immediately above the memory dies, connected via face-to-face microbumps. This approach delivers a theoretical bandwidth of over 100 TB/s for a 32 GB card, but it is not intended as a universal drop-in replacement for HBM4.
The core innovation lies in the physical arrangement: eight chiplets are stacked vertically to form a single card. Each chiplet provides 4 GB of memory and approximately 12.5 TB/s of bandwidth, totaling 32 GB and roughly 100 TB/s per card. The vertical interface consumes about 0.37 picojoules per bit, significantly lower than traditional horizontal interfaces that require long electrical traces and complex PHY circuits. This short data path is the key to reducing latency and energy consumption during high-speed data movement.
Raptor is specifically optimized for generative AI inference, particularly the token generation phase of large language models (LLMs). While the initial "prefill" stage relies heavily on compute power, the subsequent generation of tokens becomes increasingly limited by memory bandwidth. Large model weights and the Key-Value (KV) cache must be read constantly, making raw speed crucial. Raptor positions itself between SRAM, which is too expensive for large capacities, and HBM, which requires complex packaging and higher energy overhead.
Initial benchmarks presented at ISCA 2026 show promising results: the architecture achieves a 4.71x higher inference throughput compared to an HBM-based configuration and 2.44x higher than an SRAM-based one for specific models. However, these figures are not universal comparisons against every current GPU. A critical limitation is the memory capacity: with only 32 GB per card, Raptor cannot yet handle modern massive AI models that require significantly more VRAM. d-Matrix addresses this by proposing a scalable architecture using multiple cards and racks, shifting the challenge to interconnects, networking, and software management.
Thermal management is another unique constraint. The logic die is placed above the DRAM specifically because the compute layer generates more heat; stacking it on top allows for better cooling of the hotter component while power delivery must pass vertically through the memory layers. This design fundamentally changes the spatial relationship between compute and memory, offering a bandwidth that is difficult to achieve with classical interfaces.
While the 100 TB/s figure is impressive, Raptor is not a "one-size-fits-all" solution. It is a highly specialized accelerator for specific inference workloads where memory bandwidth is the primary bottleneck. As d-Matrix demonstrated with early silicon at ISCA 2026 and confirmed at Hot Chips 2026, the technology works, but its success depends on overcoming capacity limits and integrating it into larger rack-scale systems.
Гика
