Learning from Experience

Christopher Mutschler, Γεώργιος Κόντες, Sebastian RietschOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

Reinforcement Learning (RL) is a branch of Machine Learning (ML) that aims to learn from interaction with an environment. In contrast to supervised learning, where data samples are assigned ground truth labels, or unsupervised learning, which follows stationary distributions, an RL agent learns through direct interaction. This interaction defines the data collected based on the actions executed, allowing the agent to learn from experience. While traditional RL focused on continuously arriving data, recent approaches utilizing fixed data pools of past interactions have gained significant interest. This paper covers the basics of RL and discusses the latest research in interactive environments, learning with available data or knowledge, and the challenges arising from the deployment of agents to the real world.

Introduction: The Gap Between Simulation and Reality

While Reinforcement Learning has achieved spectacular success in controlled environments—most notably in games like Go and the Atari suite—applying these algorithms to physical systems like robotics, healthcare, and autonomous driving remains a formidable challenge. The fundamental gap lies between theoretical RL, which often assumes infinite, safe interactions with a perfect simulator, and the messy, constrained reality of deployment.

To bridge this gap, researchers must address four primary hurdles:

  1. Sample Efficiency: Traditional deep RL requires millions of interactions, a luxury physical hardware cannot afford.
  2. Safety: Unlike a simulation, a physical robot cannot use "trial-and-error" if an error results in catastrophic hardware failure.
  3. Data Scarcity: Many domains possess vast historical datasets but offer limited opportunities for new, active data collection.
  4. Generalization: The "Sim-to-Real" gap ensures that policies trained in idealized labs often fail when faced with real-world noise and variance.

Methodologies for Real-World RL

To move beyond simple simulations, the field has branched into several specialized paradigms designed to handle the constraints of the physical world.

The relationship between various RL paradigms and their role in achieving Real World RL.

Model-Based RL (MBRL)

Instead of learning a policy directly from raw experience, MBRL agents learn a "world model" to predict future states. This internal simulator allows for "imaginary" rollouts, enabling planning via Monte Carlo Tree Search (MCTS) or Model Predictive Control (MPC). This significantly improves sample efficiency by reducing the need for physical interactions.

Offline RL

This paradigm shifts the focus from active interaction to learning from a fixed "pool" of past data. By utilizing techniques like Conservative Q-Learning (CQL) and Decision Transformers, agents can learn optimal policies without ever touching the environment during training. A key advantage here is "stitching": the ability of an agent to take parts of different suboptimal trajectories (e.g., Path A → B and Path B → C) and combine them to find an optimal path (A → C) it never actually observed.

Safe RL

In high-stakes environments, safety cannot be a mere penalty in a reward function. Safe RL utilizes Constrained MDPs (CMDPs), treating safety as a strict mathematical constraint. By employing methods like Lagrangian multipliers, researchers ensure that agents explore the environment within predefined safety boundaries.

Key Insights: Navigating the "Deadly Triad"

A core challenge in deep RL is the "Deadly Triad": the combination of Function Approximation (Neural Networks), Bootstrapping (learning from estimates), and Off-policy training. When these three elements coexist, the learning process becomes prone to instability and divergence. Managing this triad is essential for any robust real-world application.

Furthermore, the "Noisy-TV" problem highlights a flaw in simple curiosity-driven exploration. If an agent is rewarded purely for novelty, it may become obsessed with unpredictable but useless stimuli, such as white noise on a television screen. Modern solutions like Random Network Distillation (RND) address this by rewarding the agent for predicting fixed, random features rather than raw, unpredictable pixels.

Benchmarks and Superhuman Performance

The evolution of RL is marked by several key milestones:

  • Atari 57 & Agent57: Agent57 became the first algorithm to outperform humans across the entire Atari benchmark suite, demonstrating the power of general-purpose architectures.
  • AlphaZero & MuZero: These systems demonstrated that MCTS and self-play could achieve superhuman performance in Go, Chess, and Shogi without human data (AlphaGo Zero) or even knowledge of the rules (MuZero).
  • MuJoCo: This physics engine remains the standard for testing continuous control, providing a bridge for robotics research.

Limitations and Future Directions

Despite these breakthroughs, several barriers to widespread adoption remain. Deep RL policies are often "black boxes," lacking the interpretability required for high-stakes fields like healthcare. Additionally, agents still struggle with Out-of-Distribution (OOD) scenarios; if the real world deviates slightly from the training distribution, performance often collapses.

Future research is increasingly focusing on RL from Human Feedback (RLHF) to solve the "Reward Gap"—the difficulty of designing dense, non-deceptive reward signals. As we refine these methods, the goal remains to move RL from the safety of the simulator into the complexity of the real world.

Summary updated

Abstract

Abstract Reinforcement Learning (RL) is one of the branches of Machine Learning (ML) that aims to learn from the interaction with an environment. In contrast to approaches such as supervised or unsupervised learning, where data samples usually are assigned to a ground truth label (supervised learning) or where they follow some stationary distribution (unsupervised learning), in RL, the agent is learning in direct interaction with the environment. This also defines what data is being collected as a result of which actions are being executed. The agent is hence learning from experience . While more traditionally, RL was focused purely on continuously arriving data, lately also approaches that resort to a given data pool of past environment interactions have gained more and more interest. This chapter covers the basics of RL and discusses the latest research in interactive environments, learning with available data or knowledge, and challenges that arise from the actual deployment of agents to the real world.
BibTeX
@inproceedings{mutschler2024learning,
  author = {Christopher Mutschler and Γεώργιος Κόντες and Sebastian Rietsch},
  title = {{Learning from Experience}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.1007/978-3-031-64832-8\_3},
}
Figure