Huawei Unleashes ‘KVarN’ - Boosting KV Cache by 5x Without Sacrificing Accuracy

📰 News Overview

• Staggering Capacity: Expand KV cache size by 3-5 times while maintaining FP16-level accuracy, accommodating longer contexts and numerous simultaneous requests.
• Throughput Improvement: Overcome the speed drop from quantization, achieving up to 1.3 times the throughput compared to FP16 and approximately 2.4 times compared to existing TurboQuant.
• Plug and Play: Operates as the native backend for vLLM, allowing for seamless integration without needing model changes or calibration—just a flip of a switch!

💡 Key Highlights

• Proven with Qwen3-32B: Testing with the latest model has achieved 4 times the KV cache capacity while fully maintaining FP16 accuracy.
• Hybrid Quantization: Unique configuration (k4v2) assigns 4 bits to keys and 2 bits to values, meeting stringent accuracy requirements.
• Computational Efficiency: The quantization kernel is written in Triton and optimized for runtime JIT compilation, ensuring tailored performance in various environments.

🦈 Shark’s Eye (Curator’s Perspective)

Until now, KV cache quantization has presented a dilemma: “increase capacity at the cost of speed” or “achieve speed but sacrifice accuracy.” However, KVarN tackles this challenge with a mathematically elegant approach, dispersing outliers through Hadamard rotation and minimizing quantization errors via Variance Normalization. In 2026, as agent execution and ultra-long text processing become standard, achieving five times memory efficiency while surpassing FP16 speeds signifies a true “revolution in inference”!

🚀 What’s Next?

The days of shying away from ultra-large parallel requests and processing millions of tokens due to memory constraints are over. With this technology integrated into the vLLM mainstream, we can expect a dramatic decrease in inference costs, paving the way for more affordable and high-performance AI agent services!

💬 A Word from HaruShark

Huawei’s technical prowess is sharper than a shark’s teeth, never letting prey escape! A performance boost with just a flag—there’s no reason for developers not to dive in! 🦈🔥

📚 Terminology Explained

• KV Cache: A memory area reserved for reusing past computation results during LLM generation, which expands with longer texts.

• Variance Normalization: A technique to adjust data variability and reduce information loss during quantization (the process of decreasing bit count).

• Throughput: The amount of data processed per unit time, referring to the number of tokens an AI can generate per second.

• Source: KVarN: Native vLLM backend for KV-cache quantization by Huawei