Planning through Learned Dynamics via Sample-Based Optimization

This direction is split into other directions: sampling-based model predictive control. Its last version stays readable and citable here.

Version 1 of 165 papers67 conceptsfrozen 2026-10-04

In traditional robotics, control often relies on two extremes. On one hand, classical Model Predictive Control (MPC) uses hand-engineered physics equations to plan movements, which fails when the environment is too complex to model (like soft fabric or granular sand). On the other hand, model-free Reinforcement Learning (RL) can learn anything from scratch but requires millions of trials, making it impractical for real-world hardware.

A powerful middle ground has emerged: learning a "world model" of environment dynamics using neural networks and then using sampling-based optimization to "imagine" and select the best actions. This approach allows robots to learn complex tasks—from hovering micro-robots to autonomous highway merging—with remarkably little data.

The World Model: Learning the Physics of the Task

The core of this direction is a learned dynamics model, $f_\theta(s_t, a_t) = \hat{s}_{t+1}$. Instead of deriving the laws of motion, researchers like Anusha Nagabandi and Sergey Levine show that a neural network can learn these transitions directly from raw data.

In Neural Network Dynamics Models for Control of Under-actuated Legged Millirobots, the authors demonstrated that a small neural network could learn the dynamics of a high-speed legged robot in under 17 minutes of real-world experience. This model acts as a simulator that the robot carries with it, allowing it to predict how different terrains like gravel or carpet will affect its movement.

The Shooting Method: Planning by Imagination

Once a robot has a model, how does it decide what to do? It uses a "shooting method." At every time step, the robot:

  1. Samples thousands of potential action sequences (trajectories) into the future.
  2. Rolls out these sequences through its learned dynamics model to see where they lead.
  3. Scores each predicted outcome based on a cost function (e.g., "stay close to the path" or "don't crash").
  4. Executes only the first action of the best sequence, then repeats the whole process.

This "receding horizon" strategy, detailed in Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning, is crucial. By re-planning at every step, the robot can correct for the inevitable small errors in its learned model before they accumulate into a catastrophe.

From Random Shooting to Path Integrals

Simple random sampling works for basic tasks, but complex environments require more sophisticated search. Many researchers have turned to Model Predictive Path Integral (MPPI) control. Instead of just picking the single best "shot," MPPI calculates a weighted average of all sampled trajectories, favoring those with lower costs.

This mechanism is particularly effective for systems with high-dimensional or non-linear dynamics:

Handling Uncertainty and Safety

A major risk of planning through a learned model is "model exploitation"—the optimizer finding a sequence of actions that the model thinks is good only because it has never seen those actions before.

To combat this, researchers like Yecheng Jason Ma have introduced uncertainty-aware penalties. In Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning, the planner penalizes trajectories where the model's internal uncertainty is high. Similarly, the RAZER framework by Marin Vlastelica explicitly separates "inherent noise" from "lack of knowledge," allowing the robot to explore safely while avoiding truly dangerous states Risk-Averse Zero-Order Trajectory Optimization.

Why It Works: The Power of Zero-Order Optimization

The beauty of this research direction lies in its "zero-order" nature. Because the system evaluates trajectories by sampling rather than by calculating complex gradients through the neural network, it is incredibly robust. It doesn't matter if the dynamics model is a simple MLP, a Recurrent Neural Network Data-Driven Model Predictive Control for Food-Cutting, or a massive Transformer A Generalist Dynamics Model for Control.

By decoupling the learning (the world model) from the deciding (the sampling-based planner), these systems achieve a level of flexibility and sample efficiency that allows AI to move out of the simulator and into the messy, unpredictable real world.

Papers in this version

Showing 60 of 65 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_planning_through_learned_dynamics_via_sa_fa57c32a,
  title = {Planning through Learned Dynamics via Sample-Based Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/planning-through-learned-dynamics-via-sample-based-optimization/rdr_fa57c32a3972450aa3ee8a63434013b2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_planning_through_learned_dynamics_via_sa_fa57c32a_v1,
  title = {Planning through Learned Dynamics via Sample-Based Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/planning-through-learned-dynamics-via-sample-based-optimization/rdr_fa57c32a3972450aa3ee8a63434013b2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}