$8 Chip Powers a 28.9M LLM! Google's "Per-Layer Embeddings" Leads to 100x Evolution

#ESP32-S3 #Edge AI #Gemma

※この記事はアフィリエイト広告を含みます

$8 Chip Powers a 28.9M LLM! Google’s “Per-Layer Embeddings” Leads to 100x Evolution

What Happened? A Quick Overview

LLM Execution on a Budget: A microcontroller, the $8 ESP32-S3, has successfully been used to run a language model (LLM) with 28.9 million parameters.
Astonishing Parameter Boost: This setup blows past the previous record of 260k parameters found in similar chips, achieving over 100 times that amount while being streamlined to just 14.9MB through 4-bit quantization.
Fully Offline Operation: The device generates text (stories) at a speed of about 9.5 tokens per second without relying on any external servers, displaying the output on a connected screen.

Why Does This Matter? Key Takeaways

Breaking Memory Constraints: Despite having only a tiny 512KB of SRAM, this chip applies Google’s “Per-Layer Embeddings” technology from the Gemma 3n and 4 models. By placing most of the model in slower Flash memory, it makes execution feasible.
Efficient Data Access: Instead of loading the entire Flash memory during inference, the implementation samples only the necessary lines (around 450 bytes) at a time, cleverly avoiding memory shortage issues.

🦈 Shark’s Eye (Curator’s Perspective)

This news transcends the typical narrative of “AI running on a tiny chip”! The technical prowess demonstrated here is incredible—embedding the “Per-Layer Embeddings” architecture found in the latest Gemma models into the memory layout of an $8 microcontroller, rather than a smartphone or GPU, is nothing short of revolutionary. Traditionally, these microcontrollers had to accommodate the core inference parts within SRAM, typically capping at around 260k parameters. However, by offloading most of the model weights to Flash, a “massive warehouse,” and retrieving only what’s needed, this innovation pushes the limits to a staggering 28.9 million parameters. It truly marks a revolution in Edge AI! Although the generated content is limited to simple stories due to training on the TinyStories dataset, this design philosophy of “running a massive model on compact hardware” symbolizes the future of distributed AI!

What’s Next?

With this method established, we’re headed toward a future where “not-so-smart but specialized massive models” can be integrated into everyday appliances and budget gadgets. A surge of privacy-focused devices that don’t rely on the cloud is on the horizon!

Final Thought from HaruShark

If stories can be spun from an $8 chip, I’d love to put an AI on every buoy in the ocean to record all the conversations of sharks! Sharky shark! 🦈🔥

Terminology Explained

ESP32-S3: A low-cost, low-power microcontroller equipped with Wi-Fi/Bluetooth capabilities, widely loved for DIY and IoT development.
Per-Layer Embeddings: A technique used in Google’s Gemma series for efficiently managing model weight data, well-suited for handling large models with limited computational resources.
4-bit Quantization: A technique that reduces the numerical precision of AI models, dramatically cutting computational load and memory usage, effectively shrinking model sizes by more than half.
Source: Running a 28.9M parameter LLM on an $8 microcontroller