TL;DR
Quantizing world models to low bit-widths (like 4-bit) often causes robot planning to collapse, but not for the reasons you might think. This paper reveals that iterative planners "exploit" the persistent, deterministic errors created by standard rounding; by switching to a shared stochastic dither that refreshes every iteration, researchers recovered nearly all lost performance while achieving up to 3.3× speedups.
What Problem Does This Paper Address?
In model-based reinforcement learning, a robot uses a "world model" to imagine future outcomes and pick the best actions. Sampling-based planners like MPPI (Model Predictive Path Integral) or CEM (Cross-Entropy Method) are computationally hungry, querying these models thousands of times for every single control step.
To run these models efficiently on edge devices or robots, researchers often turn to quantization—reducing the precision of neural network activations to 4 or 8 bits. However, "round-to-nearest" quantization frequently leads to "planning collapse," where the robot's success rate plummets. Curiously, this collapse happens even when the model's one-step prediction error remains low. The community lacked a clear understanding of why low-bit models fail in closed-loop planning when they seem fine in offline benchmarks.
Methods
The authors identify the culprit as deterministic error persistence. When you use standard round-to-nearest quantization, the rounding error for a specific input is always the same. Iterative planners, which refine their action choices over several rounds, essentially "search" for these systematic errors. They drift toward action sequences that the quantized model incorrectly thinks are high-reward.
To fix this without expensive retraining or calibration, the authors propose sh-iter (shared dither per iteration):
- Dithering (Stochastic Rounding): Instead of rounding to the nearest value, they round up or down stochastically. This makes the error unbiased.
- Sharing: To keep the comparison between different action candidates "fair" within a single planning iteration, they use the same random dither draw for all candidates (a technique known as Common Random Numbers).
- Redrawing: Crucially, they redraw the random dither at every iteration. This "refreshes" the error landscape, preventing the planner from latching onto and exploiting a persistent bias.
Datasets
The researchers evaluated their approach across several popular robotics benchmarks:
- Meta-World: 8 diverse manipulation tasks (e.g., button press, peg insert).
- DeepMind Control Suite (DMC): 4 continuous control tasks (e.g., walker, cheetah).
- PushT, Wall, and PointMaze: Used to test the DINO-WM model family.
- Models: They tested both the TD-MPC2 (5M and 317M parameters) and DINO-WM architectures.
Key Insights and Ideas
- Error Magnitude vs. Persistence: The most striking finding is that the size of the rounding error doesn't matter as much as its predictability. As shown in Figure 1, deterministic rounding (
det) often has a smaller one-step error than the dithered version (sh-iter), yet it performs significantly worse because the planner can "game" the deterministic errors.
- The Optimizer's Curse: The paper frames this as a manifestation of the "optimizer's curse." If a model has stationary noise (deterministic rounding error), an optimizer will naturally select for the "lucky" noise that overestimates performance.
- Drift as a Metric: The authors show that under deterministic rounding, the planner's population of candidates "drifts" toward favorable errors over iterations. Refreshing the dither kills this drift. Figure 2 demonstrates that the benefit of
sh-iteractually grows with the number of planner iterations.
Key Results
- Performance Recovery: In TD-MPC2,
sh-iterremoved at least half the planning damage in all 19 affected Meta-World task/bit-width combinations, with a median recovery of 95%. - Large-Scale Success: The method held up on a 317M-parameter multi-task model, removing 75–104% of the damage across all 8 tested tasks.
- Hardware Speedups: Using W8A4 (8-bit weights, 4-bit activations) integer kernels on an NVIDIA Ampere GPU, world model rollouts were 1.9–3.3× faster than FP32.
- Low Overhead: Drawing the random dither on the GPU is highly efficient, adding at most 2.3% to the rollout latency.
Limitations and Future Work
The authors note that sh-iter is less effective on the DeepMind Control Suite (DMC) compared to Meta-World, suggesting that the nature of "exploitable" error might vary by task domain. Additionally, the fix specifically requires an iterating planner; it provides no benefit for single-shot decision-making.
Future directions include:
- Testing on physical robot hardware.
- Porting integer kernels to other model types like DINO-WM.
- Exploring the interaction between dithering and newer hardware-level formats like NVFP4 (NVIDIA's 4-bit floating point).
- Developing better diagnostics to flag "damaging" quantization configurations before deployment.
Summary updated