Dramatically Speeding Up Million-Token Training! The World’s First Open-Source ‘Flash-MSA’ Unleashed for Hopper/Blackwell!

What’s Happening? News Overview

• The World’s First High-Performance Open-Source Training Kernel: The kernel known as “Flash-MSA” has been released, allowing efficient training of MiniMax Sparse Attention (MSA) on Hopper (H100) and Blackwell (B200) GPUs.
• Block-Level Sparsity: Utilizing the maximum value of the proxy score, KV is selected in 128-token blocks. This significantly improves memory caching characteristics.
• Adoption of GQA (Grouped-Query Attention): By adopting GQA instead of MLA (Multi-head Latent Attention), compatibility with many cutting-edge models is ensured, resulting in faster training.

Why Does This Matter? Key Points to Note

• Dramatic Reduction in Computational Load: The entire training step, aside from the proxy forward, is designed to be linear or sparse in computation relative to the context length.
• Optimized for Hopper/Blackwell: Written using CuTeDSL, it is designed to fully leverage the registers and shared memory of the latest GPUs.
• Fused Backward: By fusing the backward passes of the proxy and main attention, essential information for training signals is efficiently retrieved. This facilitates the efficient learning of sparse attention, which has been challenging until now.

🦈 Shark’s Eye (Curator’s Perspective)

Finally, a revolution in “training” for the million-token era has arrived! While the focus has been on speeding up inference (MSA and DSA), no open-source kernel has managed to achieve such high efficiency in training until now. It’s particularly astute to base this on GQA rather than MLA, allowing many existing advanced models to directly benefit. The implementation skill that harnesses Blackwell’s potential using CuTeDSL while reducing the backward computational load to linear levels is truly impressive. This is indeed a “sharp tooth” that accelerates the democratization of massive context models!

What’s Next?

With the emergence of this kernel, the additional training and fine-tuning of “super-long context models,” which previously required massive computational resources, will now be possible with smaller compute resources. Expect a wave of new open-source models adopting MSA to roll out, making million-token training the standard by 2026.

A Word from Haru Shark

With this Flash-MSA, even the rough waves of a million tokens can be conquered in one smooth swim! This relentless pace of evolution is absolutely exhilarating! 🦈🔥

Terminology Explained

• MSA (MiniMax Sparse Attention): A sparse attention technique that reduces computational load while handling long texts by selecting important KV (Key-Value) pairs in block units.

• CuTeDSL: A specialized description language that allows for highly optimized tensor operations and memory hierarchy control for NVIDIA’s Hopper/Blackwell architectures.

• GQA (Grouped-Query Attention): An attention method that shares KV among multiple queries, saving memory bandwidth while maintaining accuracy.

• Source: Flash-MSA: Accelerating Million-Token Training with Sparse Attention Kernels