Model-Based Action Exploration for Learning Dynamic Motion Skills

Glen Berseth, Alex Kyriazis, Ivan Zinin, William Choi, Michiel van de Panne

Abstract

Reinforcement Learning (RL) in continuous control often struggles with inefficient exploration in high-dimensional spaces, typically relying on stochastic noise that fails to scale with robotic complexity. We present Model-Based Action Exploration (MBAE), a framework that enables agents to perform "mental rehearsal" before executing actions. By leveraging a learned dynamics model and value function gradients, MBAE refines exploratory actions to target high-value states. Experimental results demonstrate up to a 5x speedup in learning complex motion skills and successful deployment on physical robotic hardware.

Introduction

The "curse of dimensionality" remains a significant hurdle for reinforcement learning in robotics. Standard exploration techniques, such as adding Gaussian noise to policy outputs, become increasingly ineffective as the degrees of freedom grow. In high-dimensional spaces, random perturbations are unlikely to yield meaningful improvements, often resulting in jerky or dangerous movements on physical hardware. To address this, we propose MBAE, which moves away from blind exploration toward an informed, look-ahead approach where the agent uses its current understanding of physics to try actions that seem promising.

Methodology

The core of MBAE is an internal look-ahead mechanism that allows the agent to evaluate the potential outcomes of its actions before they are executed in the environment.

1. Dynamics Modeling

The agent learns a transition probability model $\hat{P}(x_{t+1} | x_t, u_t)$ using a Conditional Generative Adversarial Network (cGAN). Unlike traditional Mean Squared Error (MSE) models that tend to average out outcomes in complex environments, the cGAN captures the inherent stochasticity of physics, allowing the agent to sample distinct, realistic future states.

2. Action Refinement via Gradients

When exploring, the agent passes its intended action through the dynamics model to predict a future state. It then evaluates this state using its current Value Function ($V$). Because both the dynamics model and the value function are differentiable neural networks, the agent can compute the gradient of the predicted state's value with respect to the input action. By taking a small step in the direction of this gradient, the agent "improves" its exploratory action to favor transitions into high-value regions of the state space.

The MBAE framework architecture showing the integration of policy, dynamics model, and value function for action refinement.

Experimental Setup

The MBAE framework was evaluated across several benchmarks, ranging from synthetic navigation tasks to complex robotic simulations and physical hardware:

  • N-Dimensional Particle Navigation: Testing scalability in high-dimensional grid-worlds.
  • 2D Biped Imitation: Learning walking gaits from reference motions.
  • OpenAI Gym: Standard benchmarks including HalfCheetah and 2D Reacher.
  • The Membrane Robot: A custom physical platform consisting of parallel links that create a wave-like surface for tasks such as ball juggling and box stacking.
The physical Membrane Robot setup used to validate MBAE in real-world scenarios.

Results and Discussion

MBAE significantly outperformed baseline methods in both learning speed and final policy quality. In the 2D Biped walking task, the agent reached proficiency five times faster than the baseline CACLA algorithm. On the HalfCheetah benchmark, MBAE achieved a final reward three times higher than the baseline.

Crucially, the method proved efficient enough for real-world robotics. A "move-to-target" policy was successfully trained on the physical Membrane Robot in just three hours. The results suggest that even an imperfect dynamics model can provide useful gradients for exploration, acting as a stabilizer that filters out noisy updates.

Conclusion

Model-Based Action Exploration provides a principled way to incorporate "thinking" into the exploration process of RL agents. By utilizing learned models to guide action selection, MBAE overcomes the limitations of random noise, offering a path toward more efficient and safer learning in high-dimensional robotic systems.

Summary updated

Abstract

Deep reinforcement learning has achieved great strides in solving challenging motion control tasks. Recently, there has been significant work on methods for exploiting the data gathered during training, but there has been less work on how to best generate the data to learn from. For continuous action domains, the most common method for generating exploratory actions involves sampling from a Gaussian distribution centred around the mean action output by a policy. Although these methods can be quite capable, they do not scale well with the dimensionality of the action space, and can be dangerous to apply on hardware. We consider learning a forward dynamics model to predict the result, (x <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t+1</sub> ), of taking a particular action, (u), given a specific observation of the state, (x <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> ). With this model we perform internal lookahead predictions of outcomes and seek actions we believe have a reasonable chance of success. This method alters the exploratory action space, thereby increasing learning speed and enables higher quality solutions to difficult problems, such as robotic locomotion and juggling.
BibTeX
@inproceedings{berseth2018model,
  author = {Glen Berseth and Alex Kyriazis and Ivan Zinin and William Choi and Michiel van de Panne},
  title = {{Model-Based Action Exploration for Learning Dynamic Motion Skills}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  doi = {10.1109/iros.2018.8593588},
}
Figure