In traditional robotics, control often relies on two extremes. On one hand, classical Model Predictive Control (MPC) uses hand-engineered physics equations to plan movements, which fails when the environment is too complex to model (like soft fabric or granular sand). On the other hand, model-free Reinforcement Learning (RL) can learn anything from scratch but requires millions of trials, making it impractical for real-world hardware.
A powerful middle ground has emerged: learning a "world model" of environment dynamics using neural networks and then using sampling-based optimization to "imagine" and select the best actions. This approach allows robots to learn complex tasks—from hovering micro-robots to autonomous highway merging—with remarkably little data.
The World Model: Learning the Physics of the Task
The core of this direction is a learned dynamics model, $f_\theta(s_t, a_t) = \hat{s}_{t+1}$. Instead of deriving the laws of motion, researchers like Anusha Nagabandi and Sergey Levine show that a neural network can learn these transitions directly from raw data.
In Neural Network Dynamics Models for Control of Under-actuated Legged Millirobots, the authors demonstrated that a small neural network could learn the dynamics of a high-speed legged robot in under 17 minutes of real-world experience. This model acts as a simulator that the robot carries with it, allowing it to predict how different terrains like gravel or carpet will affect its movement.
The Shooting Method: Planning by Imagination
Once a robot has a model, how does it decide what to do? It uses a "shooting method." At every time step, the robot:
- Samples thousands of potential action sequences (trajectories) into the future.
- Rolls out these sequences through its learned dynamics model to see where they lead.
- Scores each predicted outcome based on a cost function (e.g., "stay close to the path" or "don't crash").
- Executes only the first action of the best sequence, then repeats the whole process.
This "receding horizon" strategy, detailed in Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning, is crucial. By re-planning at every step, the robot can correct for the inevitable small errors in its learned model before they accumulate into a catastrophe.
From Random Shooting to Path Integrals
Simple random sampling works for basic tasks, but complex environments require more sophisticated search. Many researchers have turned to Model Predictive Path Integral (MPPI) control. Instead of just picking the single best "shot," MPPI calculates a weighted average of all sampled trajectories, favoring those with lower costs.
This mechanism is particularly effective for systems with high-dimensional or non-linear dynamics:
- Soft Robotics: Mark Van der Merwe and colleagues used MPPI to control compliant tools like spatulas, where the tool's bending makes traditional modeling impossible Learning the Dynamics of Compliant Tool-Environment Interaction for Visuo-Tactile Contact Servoing.
- Deformable Objects: In DYMO-Hair, Chengyang Zhao used MPPI rollouts to "imagine" how combing actions would deform 3D hairstyles.
- Autonomous Driving: Minjun Sung developed IANN-MPPI, which uses sampling to "negotiate" with other drivers by predicting their reactions to the ego-vehicle's potential maneuvers.
Handling Uncertainty and Safety
A major risk of planning through a learned model is "model exploitation"—the optimizer finding a sequence of actions that the model thinks is good only because it has never seen those actions before.
To combat this, researchers like Yecheng Jason Ma have introduced uncertainty-aware penalties. In Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning, the planner penalizes trajectories where the model's internal uncertainty is high. Similarly, the RAZER framework by Marin Vlastelica explicitly separates "inherent noise" from "lack of knowledge," allowing the robot to explore safely while avoiding truly dangerous states Risk-Averse Zero-Order Trajectory Optimization.
Why It Works: The Power of Zero-Order Optimization
The beauty of this research direction lies in its "zero-order" nature. Because the system evaluates trajectories by sampling rather than by calculating complex gradients through the neural network, it is incredibly robust. It doesn't matter if the dynamics model is a simple MLP, a Recurrent Neural Network Data-Driven Model Predictive Control for Food-Cutting, or a massive Transformer A Generalist Dynamics Model for Control.
By decoupling the learning (the world model) from the deciding (the sampling-based planner), these systems achieve a level of flexibility and sample efficiency that allows AI to move out of the simulator and into the messy, unpredictable real world.