Reinforcement learning (RL) has a "creativity" problem. When tasked with moving a robot from point A to point B, an RL agent often discovers bizarre, jittery, or physically inefficient gaits that exploit simulation inaccuracies—patterns that would never work on real hardware. To solve this, a growing body of research is moving away from "pure" black-box learning. Instead, researchers are using physics-based models and trajectory optimization as a "scaffold" to guide the learning process.
By providing the agent with a physical prior—whether it’s a simplified spring model or a pre-optimized trajectory—we can produce policies that are both robust and physically grounded.
Trajectory Optimization as a Behavioral Prior
The most direct way to guide an RL agent is to show it what a physically feasible movement looks like. However, raw data (like human motion capture) is often "physically inconsistent" when applied to a robot with different proportions.
To bridge this "embodiment gap," researchers like Victor Dhedin and Ilyass Taouil developed DynaRetarget. This framework uses sampling-based trajectory optimization to "clean" human data, ensuring the resulting reference motions respect the robot's dynamics. Similarly, Ri-Zhao Qiu and Xiaolong Wang introduced AMO, which uses dynamics-aware optimization to generate reference data for humanoid whole-body control.
The intuition here is that RL shouldn't have to "reinvent the wheel" of physics. By starting with a trajectory that is already physically plausible, the agent can focus its exploration on handling perturbations and noise rather than struggling to stand up. As Lujie Yang and Guanya Shi demonstrate in OmniRetarget, high-fidelity reference motions drastically simplify reward engineering; if the reference is good, the agent only needs a few basic rewards to learn complex interactions.
Simplified Templates for Complex Dynamics
Sometimes, a full trajectory is too rigid. Instead, we can guide RL using "templates"—simplified physics models that capture the essence of a movement without dictating every joint angle.
One classic template is the Spring-Loaded Inverted Pendulum (SLIP). Daniel Felipe Ordoñez Apraez and Mario Martin showed that by biasing RL toward SLIP dynamics, robots naturally develop energy-efficient, biological-looking gaits like trotting or running without any motion capture data.
This "template" approach can also be used to define how a robot feels to the touch. In the FACET framework, Qingzhou Lu and Huazhe Xu use a virtual mass-spring-damper system as a reference. The RL agent is rewarded for matching the center-of-mass motion of this virtual model, allowing the robot to seamlessly switch between being soft and compliant or stiff and powerful.
Heuristics and Interaction Chains
When robots interact with objects, the physics becomes even more complex. Joanne Truong and Alexander Clegg address this in RobotMover by focusing on the "Interaction Chain." Rather than imitating exact human limb movements, the robot imitates the path of the object and the forces applied to it. This morphology-agnostic approach allows a quadruped to move furniture by understanding the physics of the task rather than the kinematics of the human.
For simpler locomotion, explicit heuristics can replace data entirely. The Ask1 quadruped, developed by Yang Zhang and colleagues, uses classical Raibert Heuristics to calculate ideal foot placements. By rewarding the agent for landing near these calculated points, the robot learns stable, agile locomotion on rugged terrain without needing any pre-recorded animations.
Why This Hybrid Approach Works
The recurring theme across these papers—from VERONICA by Simiao Zuo and Ye Zhao 0002 to the loco-manipulation work of Renjie Wang—is a division of labor.
- Physics/Optimization handles the "what": what is physically possible, what is energy-efficient, and what respects the laws of gravity.
- Reinforcement Learning handles the "how": how to map sensor inputs to motor torques, how to recover from a slip, and how to generalize to a messy, unpredictable world.
By grounding RL in physical reality, we move from robots that "hack" their simulations to robots that understand the fundamental dynamics of movement.