TL;DR
WM-Quant is the first post-training quantization (PTQ) framework specifically designed for JEPA-style world models used in robotic planning. By addressing the "compounding error" problem inherent in autoregressive rollouts, it enables 4-bit (W4A4) models to achieve near-full-precision success rates with a ~1.7x speedup on edge hardware like the NVIDIA Jetson.
What Problem Does This Paper Address?
World models are internal simulators that allow robots to "imagine" the consequences of their actions before taking them. Using Joint Embedding Predictive Architectures (JEPA), these models reason in an abstract latent space rather than pixels, making them much faster. However, real-time planning still requires thousands of model inferences per decision (using Model Predictive Control), creating a massive computational bottleneck for edge devices.
Standard quantization—reducing the precision of weights and activations to 4 bits—is the go-to solution for speed. But in world models, it fails spectacularly. Because these models are autoregressive (the output of one step is the input for the next), tiny quantization errors compound at every step of a "rollout." By the end of a 5-step prediction, the error is so large that the model can no longer distinguish a good action from a bad one, leading to near-random behavior.
Methods
The authors introduce WM-Quant, a three-pronged approach to stabilize low-precision world models:
- Early-Encoder Precision: The encoder creates the initial "map" for the rollout. If this is noisy, the entire plan is doomed. WM-Quant keeps the activations of the first three encoder blocks at 8-bit precision while quantizing the rest of the model to 4-bit.
- Control-Path Preservation: Action embeddings are often small in magnitude compared to state embeddings. Quantizing them can cause "scale collapse," where the model loses the ability to differentiate between similar actions. WM-Quant keeps the action-conditioning path in floating point (FP), which only affects ~1.5–7.5% of total parameters.
- Rollout-Aware Scaling: Instead of calibrating quantization ranges for a single prediction step, WM-Quant calibrates them across an entire multi-step rollout. It also assigns unique activation ranges to each step in the rollout to account for shifting data distributions as the "imagination" progresses.
Datasets
The framework was evaluated on two state-of-the-art JEPA world models (LeWM and DINO-WM) across several benchmarks:
- PushT: A contact-based robotic manipulation task.
- TwoRoom: A simple navigation task.
- Wall: A navigation task involving obstacles.
Key Insights and Ideas
- The "LLM Reset" Doesn't Exist Here: In Large Language Models, the model "snaps" its continuous output to a discrete token at each step, which can effectively reset small errors. JEPA models work entirely in continuous latent space; there is no discrete "reset," making them far more sensitive to compounding noise.
- Action Rankings are Fragile: The paper shows that even if a quantized model's raw MSE (Mean Squared Error) looks okay, the ranking of actions can be totally scrambled. If the top-ranked action in full precision becomes the 50th-ranked action in 4-bit, the robot fails. (See Figure 4).
- Dynamic Range Shifting: Activation distributions change as a rollout progresses. Using a single static clipping range for all steps (the standard PTQ approach) introduces unnecessary clipping or rounding errors.
Key Results
WM-Quant significantly outperforms standard PTQ methods (like SmoothQuant or DuQuant) across the board:
- Success Rates: On the LeWM PushT task at W4A4 precision, standard methods achieved only ~20% success. WM-Quant boosted this to 81.3%, approaching the 98% success of the full-precision (FP16) model.
- Extreme Quantization: Even at W3A3 (3-bit), WM-Quant achieved a 58% success rate on PushT, whereas baselines hovered around 10%.
- Hardware Efficiency: When deployed on an NVIDIA Jetson AGX Orin, WM-Quant provided a 1.7x speedup over FP16 with negligible memory overhead, making real-time, low-power planning viable.
Limitations and Future Work
- Heuristic Bit-Width Selection: The choice to keep exactly three encoder blocks at 8-bit is a heuristic. Future work could use automated mixed-precision searches to find optimal bit-budgets for specific hardware.
- Manual Control-Path Handling: While keeping the control path in FP is effective, it requires manual identification of those layers. A more automated "sensitivity-aware" quantization process could streamline this.
- Architecture Scope: The study focused on JEPA-style models. Exploring how these rollout-aware techniques apply to other world model architectures (like the reconstruction-based Dreamer series) remains an open question.
Summary updated