In the world of robot learning, the prevailing wisdom is often "more is better." More data, more parameters, and certainly more planning. If your robot isn't solving a task, the intuitive fix is to give it more "thinking time"—more optimization steps to find a better plan.
A new paper by Josef Chen, "When More Planning Hurts: Learned Planners for World Models," challenges this fundamental assumption. Through a series of rigorous experiments on a visual manipulation task, Chen demonstrates a counterintuitive reality: in the presence of imperfect world models, extra planning effort can actually lead to worse performance.
TL;DR
This paper investigates the relationship between planning effort and task success in latent world models. Surprisingly, it finds that increasing the number of optimization updates can sometimes turn a successful robot trajectory into a failure, even when the planner successfully finds a "better" (lower-cost) path according to the model.
What Problem Does This Paper Address?
Model-based reinforcement learning relies on a "world model" to predict the outcomes of actions. A planner then searches this model to find a sequence of actions that minimizes a cost function. However, these models are never perfect.
The authors tackle the objective mismatch problem: the gap between what the model thinks is a good plan (low predicted cost) and what actually works in the real world (task success). Specifically, they ask: does spending more computational effort to optimize a plan within a learned world model always yield better results at execution time?
Methods
The study focuses on the PushT task, a simulated visual manipulation challenge where a robot must push a T-shaped block into a target pose.
The researchers compared several planning approaches:
- Fixed Action Updater: A tiny, supervised neural network (only 1,505 parameters) designed to refine plans.
- Conventional Optimizers: Standard methods like Adam and iCEM (improved Cross-Entropy Method).
- LePlanner: A previously released, high-performance learned controller.
The core of the paper is a matched intervention study. Starting from the exact same state and history, the authors varied only the "planning budget"—the number of updates allowed (4, 8, or 12). This allowed them to isolate the effect of planning effort on the final outcome.
Datasets
- PushT: The primary simulation environment for visual manipulation.
- Reserved Scenes: A set of 16 specific starting configurations (scenes) used for the final "confirmation" evaluation to ensure the results weren't just a fluke of the training data.
- Development Scenes: An initial set of 24 scenes used for tuning and preliminary experiments.
Key Insights and Ideas
- The "More Planning Hurts" Phenomenon: In a matched experiment, increasing the planning budget from 4 to 12 updates actually decreased the success rate. Tasks that were solved with 4 updates became failures with 12.
- Optimization Success vs. Execution Failure: The planners weren't "failing" in the traditional sense; they were successfully finding plans with lower predicted costs. However, these "better" plans led the robot into regions where the world model's inaccuracies caused the task to fail.
- The Hidden Danger of Lower Scores: Figure 1 perfectly illustrates this. A plan with 12 updates achieved a much lower latent score (10.294 vs 19.676) but failed because it misjudged the required pusher position, leading to a massive joint error.
- Model Inaccuracy as a Ceiling: The quality of the world model acts as a hard ceiling. If the model is wrong about the physics of a certain interaction, no amount of planning "inside" that model will fix the resulting real-world error. In fact, aggressive optimization might just exploit the model's flaws more effectively.
Key Results
The quantitative results highlight the struggle of the learned updater against established baselines:
- Success Rates (out of 16 scenes):
- LePlanner: 16/16 (The gold standard)
- Learned Updater (8 updates): 8/16
- Adam: 7/16
- iCEM: 2/16
- The Planning Effort Trade-off: In the matched experiment, the success counts were 9/16 for 4 updates, 8/16 for 8 updates, and 7/16 for 12 updates. Extra effort rescued zero failing tasks but caused two previously successful ones to fail.
- Efficiency: While the learned updater didn't beat LePlanner in success, it was significantly faster than Adam, using only about 27% of the planning and encoding time.
Limitations and Future Work
The author is transparent about several limitations:
- Scope: The findings are based on a single task (PushT) and a relatively small number of scenes.
- Architecture: The study doesn't fully isolate the effect of the controller architecture from other factors like training data and objectives.
- Generalization: It's unclear if these results hold for different model families or real-world robotic hardware.
Future directions include testing these ideas on a wider variety of tasks, exploring real-robot deployments, and investigating whether "cheap" contextual policies (knowing when to stop planning) can mitigate the observed harms of over-optimization.
Summary updated