Deep reinforcement learning (RL) has achieved remarkable success in simulated environments, often consuming billions of frames to master complex behaviors Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes. However, when moving from simulation to physical hardware, algorithms hit a hard wall of physical reality. Physical robots cannot run faster than real time, they suffer from mechanical wear, they run out of battery, and they risk catastrophic structural failure during erratic exploration Learning Flexible and Reusable Locomotion Primitives for a Microrobot.
This reality imposes a strict constraint: to be practical, robots must acquire skills—such as folding fabric, walking, or manipulating objects—in a matter of minutes or hours, using at most a few hundred trials Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience. Achieving this level of sample efficiency requires moving away from tabula rasa model-free exploration. Instead, modern approaches inject geometric, physical, and structural priors into the learning loop, shifting the sample burden away from the physical world.
Constraining the Search Space via Structured Policies
If a robot explores a high-dimensional continuous action space completely at random, it will spend hours executing useless or damaging movements. We can drastically accelerate learning by shaping the policy's optimization landscape or restricting its action space to stable primitives.
One foundational approach is to respect the underlying geometry of the control policy. Jan Peters, Jan Peters 0001, and Stefan Schaal demonstrated that using natural gradients—which account for the Riemannian geometry of the policy parameter space rather than treating it as a flat Euclidean space—prevents destructive policy updates and enables a robot arm to learn complex motor primitives in a few hundred trials Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning.
Another strategy is to guide global neural network exploration using local trajectory optimization. In Learning Contact-Rich Manipulation Skills with Guided Policy Search, Levine et al. utilize physics-based local controllers to guide a global policy. This prevents the robot from flailing blindly in contact-rich environments, reducing the required real-world interaction time to minutes.
Alternatively, the action space itself can be restricted. Roberto Calandra and Tianyu Li show that planning within a learned, low-dimensional latent action space makes legged locomotion highly sample-efficient and robust to hardware failures Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion. For quadruped robots, Zhang et al. map actions to bio-inspired Central Pattern Generators (CPGs), allowing the system to optimize locomotion parameters online in under three minutes by searching only over stable, rhythmic gait variations Online Optimization of Central Pattern Generators for Quadruped Locomotion.
Shifting the Sample Burden to Internal World Models
Model-based reinforcement learning (MBRL) improves sample efficiency by training a predictive "world model" on real-world data. The robot can then dream up millions of trajectories in its internal simulator, executing actions on physical hardware only when necessary.
Keller et al. leverage this via Model-Based Quality-Diversity (M-QD), using a neural network to predict action outcomes and evaluate behavioral diversity Model-Based Quality-Diversity Search for Efficient Robot Learning. This allows the algorithm to filter out unpromising behaviors entirely in imagination, protecting physical hardware. Grillotti et al. extend this paradigm to autonomous skill discovery, demonstrating that combining world models with Quality-Diversity algorithms allows a quadruped to discover a wide library of agile gaits directly on hardware within five hours From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity.
[Physical Interaction] ---> [Update World Model] ---> [Differentiable Physics/Rendering]
^ |
| v
[Execute Policy] <--- [Optimize Policy via Imagined Rollouts] <--+
To build accurate world models directly from visual inputs, Lv et al. introduce SAM-RL, which incorporates differentiable physics-based simulation and rendering SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering. By computing analytical gradients directly through the physics engine and renderer, the robot updates its internal model and selects informative camera viewpoints with a fraction of the data required by standard deep RL.
When onboard computation introduces latency, Zakariae El Asri demonstrates that combining a physics-informed model with a multi-step planning strategy can mitigate inference delays while maintaining high sample efficiency RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms. Furthermore, learned models can be paired with classical control: Karnchanachari et al. combine RL with Model Predictive Control (MPC) to safely learn value functions from scratch under sparse objectives in under an hour Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot.
Reusing Data and Localizing Optimization
A robot should not start from scratch for every new task. Reusing prior data or narrowing the optimization focus to local regions can drastically accelerate convergence.
Hu et al. introduce the REBOOT framework, which pre-loads reinforcement learning replay buffers with data from previous tasks and different objects, cutting real-world dexterous manipulation training times in half REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation. Similarly, Zhan et al. combine a small set of human demonstrations with unsupervised contrastive pre-training and data augmentation to master manipulation tasks from raw pixels in under 30 minutes A Framework for Efficient Robotic Manipulation. Even in chaotic, unmodelable environments like fluid dynamics, Bhardwaj et al. show that offline RL can effectively repurpose existing interaction data to learn complex object manipulation without requiring continuous physical operation of the system Real-world fluid directed rigid body control via deep reinforcement learning.
When global optimization is too expensive, localizing the search space provides a massive efficiency boost. Riad Akrour and Dmitry Sorokin introduce L-BayesOpt, which restricts Bayesian Optimization to a moving Gaussian distribution Local Bayesian Optimization of Motor Skills. This combines the sample efficiency of Bayesian methods with the high-dimensional scalability needed for motor skills. This sample efficiency is critical for platforms where extensive exploration is impossible, such as modular robots with evolvable morphologies Learning Directed Locomotion in Modular Robots with Evolvable Morphologies.
Safe Exploration and Rapid Adaptation
Real-world deployment requires the robot to handle physical constraints safely and adapt to unexpected changes without triggering a costly retraining loop.
In human-robot interaction scenarios, such as assistive dressing, end-to-end exploration is too risky. Sun et al. address this in Force-Constrained Visual Policy: Safe Robot-Assisted Dressing via Multi-Modal Sensing by pairing a simulation-trained visual policy with a force-dynamics model learned directly from real-world interactions, ensuring strict safety boundaries.
When a robot suffers physical damage in the field, it must adapt on the fly. Chatzilygeroudis et al. propose the Semi-Episodic Learning Algorithm (SELA), which models the general outcomes of movements rather than a single task score, allowing damaged robots to recover functional behaviors in a very small number of trials Towards semi-episodic learning for robot damage recovery. To handle unfamiliar objects without prior knowledge of their physical properties, Ding et al. introduce TMOC, which interleaves real-world execution with physics simulations to ground symbolic plans into physical reality through rapid trial and error Learning to Ground Objects for Robot Task and Motion Planning.
Finally, meta-learning offers a powerful paradigm for rapid adaptation. Schoettler et al. show that training a policy on a randomized family of simulated insertion tasks allows a robot to learn a structured exploration strategy, adapting to real-world industrial insertion tasks in under 20 trials Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks. Alternatively, highly optimized model-free algorithms can bypass simulation entirely: the authors of Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion leverage the CrossQ algorithm to achieve agile quadruped locomotion directly on physical hardware in just 8 minutes.
Key Foundational Work
Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning (2007) — contributed 1 concept elements to this direction.
Learning CPG Sensory Feedback with Policy Gradient for Biped Locomotion for a Full-Body Humanoid (2005) — contributed 1 concept elements to this direction.