Learning the Mario World Solo!? The World Model ‘LeMario’ Using JEPA Reveals the Borders of Prediction and Strategy

What Happened? Overview of the News

• Recreating the JEPA Architecture: Implementing Yann LeCun’s “Joint-Embedding Predictive Architecture (JEPA)” in Mario. The model learned the mechanics of the world from pixel images and button inputs without any rewards.
• Stunning Short-Term Prediction Accuracy: The trained model can accurately predict the future five steps ahead, even in unknown episodes. It successfully maneuvers Mario with pixel-level precision towards the goal.
• The Wall Between ‘Prediction’ and ‘Strategy’: While short-term movement has been achieved, the model has yet to develop long-term planning skills, such as jumping over large obstacles or navigating to distant goals.

Why Is This Important? Key Points to Note

• Self-Supervised Learning Without Rewards: Unlike traditional reinforcement learning that requires teaching a “score,” the AI autonomously understands the causal relationship of “press this button, and the screen changes.”
• Action Injection with AdaLN-Zero: Using “AdaLN-Zero” in the transformer block allows for efficient future predictions by adjusting the scale and gates of features based on actions like jumping.
• Preventing Representation Collapse: By incorporating “SIGReg” into the loss function, the project constructs a latent space that maintains diversity of information, preventing all images from appearing identical.

🦈 Shark’s Eye (Curator’s Perspective)

What’s thrilling about this project is that it’s not just about letting Mario play; it’s about implementing JEPA’s philosophy of embedding the “rules of the world” from scratch! Particularly eye-catching is the application of AdaLN-Zero. Instead of just attaching action vectors, it intelligently controls the Shift (displacement) and Scale (emphasis) of features, enabling dynamic processing like “prioritizing vertical information while jumping”—super smart!

The outcome of “being able to predict but not win the game” highlights the deep gap between AI’s “understanding of physical laws” and “achieving objectives,” making it an exhilarating research theme for 2026!

What’s Next?

With such accuracy in short-term predictions, the integration of “long-term hierarchical planning” will be crucial. The phase will challenge how to connect the “future map” drawn by the world model to the “winning scenarios,” truly testing the value of JEPA!

A Word from Haru Shark

Even if you can move Mario, taking down Bowser requires both “passion” and “long-term planning”! Become a shark with 100% prediction accuracy and devour the future! 🦈🔥

Terminology Explained

• JEPA (Joint-Embedding Predictive Architecture): An architecture that predicts the future in an abstracted “latent space” rather than directly predicting images, allowing it to ignore unnecessary noise.

• AdaLN-Zero: A technique for adjusting layers in neural networks. By initializing weights to zero, it prevents unnecessary noise from mixing in the early stages of training and gradually reflects the impact of actions.

• SIGReg: A regularization technique to prevent “representation collapse,” where latent representations converge to identical values. It is used to maintain the diversity of data.

• Source: LeMario: Training a JEPA World Model on Super Mario Bros