Breaking the Limits of Apple Silicon! Meet “Espresso,” the ANE Direct Inference Framework 4.76x Faster than CoreML

What’s the Buzz? Overview of the News

• New Tech Bypassing CoreML: The “Espresso” framework has arrived, enabling direct control of Apple Silicon’s Neural Engine (ANE) through private APIs, bypassing CoreML entirely.
• A Staggering 4.76x Speed Boost: Compared to conventional inference via CoreML, it achieves approximately 4.76 times faster decoding (1.08 ms/token).
• Support for Training as Well: Not only does it support forward passes on the ANE, but it also includes full training capabilities with backward passes (gradient accumulation, Adam optimization).

Why Is This Significant? Key Points to Note

• Success in Reverse Engineering Private APIs: By reverse-engineering internal APIs like _ANEClient and _ANEInMemoryModel, Espresso compiles and executes direct MIL (Model Intermediate Language) programs.
• Zero-Copy I/O Implementation: Utilizing IOSurface buffers for NEON-vectorized reads, it eliminates memory transfer overhead entirely.
• Latest Design with Swift 6.2: Built with state-of-the-art specifications as of 2026, featuring ~Copyable (move-only) tensors and strict concurrency checks, functioning with zero dependencies.

🦈 Shark’s Eye (Curator’s Perspective)

Finally, the “true seal” of Apple Silicon has been broken! Until now, running AI on a Mac meant sticking to CoreML or Metal, but Espresso dives directly into the black box that is ANE—talk about impressive!

The design that fuses three layers of kernels into one to minimize command calls to the ANE showcases incredible craftsmanship that understands hardware characteristics. By completely skipping the “recompilation per token” and “heavy transformation processes” that are unavoidable with CoreML, it’s no wonder there’s such a speed difference. With an M4 Max fully utilizing its 38-core ANE, we can expect a response time that feels like anything but a local LLM!

What’s Next?

The previous limitation of “ANE can only be used via CoreML” has been shattered by Espresso, paving the way for a new wave of highly optimized Apple Silicon-specific models. Especially for mobile devices and edge AI training on Mac minis, the efficiency of AI execution across Apple’s ecosystem is set to level up to the next stage!

A Word from Haru-Same

The true potential of Apple Silicon is far beyond what we thought! With Espresso taming the ANE directly, you can transform your Mac into an ultimate AI machine! Sharky shark! 🦈🔥

Terminology Explained

• ANE (Apple Neural Engine): A specialized hardware designed for AI processing embedded in Apple Silicon. It’s incredibly powerful but typically difficult for developers to control directly.

• MIL (Model Intermediate Language): An intermediate language for expressing Apple’s machine learning models. Espresso compiles this directly into binaries for the ANE.

• IOSurface: A framework that allows for fast memory sharing between different hardware (CPU, GPU, ANE, etc.) without causing data copies.

• Source: Train and run transformers directly on Apple’s Neural Engine