Abstract
Training reinforcement learning (RL) agents for complex robotic tasks often requires access to internal simulator states and millions of simulated experiences. This paper presents a five-stage framework that enables training autonomous agents from raw visual inputs in wall-clock time. By integrating imitation learning (IL), human-guided reward induction, and a safety module, the proposed system successfully trains a driving agent in the realistic Assetto Corsa environment, achieving zero-accident performance while surpassing human imitation policies.
Introduction
Standard reinforcement learning typically relies on high-speed simulators and hand-crafted reward functions derived from internal environment variables (e.g., precise coordinates). However, real-world robotics must operate in wall-clock time and often only have access to raw sensor data like pixels. Furthermore, the random exploration inherent in RL can lead to catastrophic failures in physical systems. This research addresses these challenges by leveraging human supervision to provide both "strong" guidance (demonstrations) and "weak" guidance (binary labels) to bridge the gap between raw pixels and complex task execution.
The Multi-Stage Learning Framework
The authors propose a pipeline centered on Convolutional Neural Networks (CNNs) to process raw pixel data across several distinct phases:
- Imitation Learning (IL): A policy network is trained to mimic human steering actions using supervised learning on a dataset of ~70,000 state-action pairs.
- Deep Reward Induction: Instead of manual engineering, a reward network is trained using binary human feedback. Human instructors label frames as "good" or "bad" based on specific task objectives, such as maintaining a specific lane.
- Safety Module Construction: A safety network classifies states as "safe" or "unsafe." During RL training, if a state is flagged as unsafe, the pre-trained IL policy takes over to prevent a crash.
- Reinforcement Learning (DDQN): A Double Deep Q-Network is initialized with IL weights and trained using the induced reward signal, with exploration constrained by the safety module.
Key Insights: Weak Supervision and Safety Guardrails
A critical finding is that human instructors do not need to be perfect drivers to be effective teachers. By providing "weak" binary labels, they allow the system to build robust reward and safety models. The safety module acts as an exploration guardrail, preventing the agent from wasting time in dangerous or irrelevant regions of the state space.
Furthermore, initializing the RL agent with IL weights significantly accelerates convergence. The reward network demonstrates impressive generalization, successfully mapping abstract human intents (like "stay in the second lane") directly from pixels without explicit lane-detection algorithms.
Experimental Results
The framework was evaluated in the Assetto Corsa racing simulator. The results highlight the efficiency of the integrated approach:
- Accident Reduction: During the initial training phase, a standard RL agent averaged 116.6 crashes per epoch. The agent utilizing IL initialization and the safety module reduced this to 47 crashes, eventually reaching a state of zero accidents.
- Superior Performance: The final RL agent consistently outperformed both the baseline RL model and the original human imitation policy in terms of accumulated rewards.
- Sample Efficiency: The system achieved autonomous driving capability using only 3.5 million frames, equivalent to approximately 4 days of wall-clock time, proving the feasibility of the approach for real-world applications.
Conclusion and Future Work
This work demonstrates that combining strong and weak human supervision allows for the efficient training of robotic agents from raw pixels in real-time. While current limitations include the manual labeling effort and single-variable control (steering only), future research aims to extend this to multi-controller tasks (e.g., simultaneous braking and acceleration) and explore unsupervised methods to further reduce the human labeling burden.
Summary updated