Introduction
Deep Reinforcement Learning (DRL) has demonstrated remarkable potential in complex decision-making tasks, yet it remains plagued by sample inefficiency and instability, often requiring millions of interactions to master basic behaviors. Conversely, Imitation Learning (IL) offers high data efficiency but is vulnerable to "distribution mismatch," where minor errors lead the agent into unfamiliar states from which it cannot recover.
General Reinforced Imitation (GRI) is a framework designed to bridge this gap. By treating expert demonstrations as high-reward transitions within a standard RL replay buffer, GRI provides a straightforward method to inject expert knowledge into any off-policy RL algorithm without disrupting the underlying reinforcement learning mechanics.
The GRI Framework
The core innovation of GRI is the simplifying hypothesis that expert demonstrations can be viewed as perfect data that consistently receives a constant, maximum reward ($r_{demo}$). This is implemented through Offline Demonstration Agents.
In a standard RL setup, an exploration agent interacts with the environment and stores experiences in a replay buffer. In the GRI framework, a parallel demonstration agent—essentially a specialized data loader—injects expert trajectories into the same buffer. The RL backbone then samples from this mixed buffer, treating expert and self-explored transitions identically. This approach allows the agent to learn from high-quality data while maintaining the ability to explore and refine its policy through environmental rewards.
Application to Autonomous Driving (GRIAD)
To demonstrate the efficacy of GRI in a high-stakes domain, the authors developed GRIAD (GRI for Autonomous Driving). The system utilizes a two-phase pipeline:
- Perception Pre-training: Visual encoders (EfficientNet-b1) are trained on auxiliary tasks, including semantic segmentation and traffic light classification, to extract robust features from three camera views (Center, Left, Right).
- Decision Learning: The encoders are frozen, and a distributed DRL backbone (Rainbow-IQN Ape-X) is trained using the GRI method.
Experimental Setup and Datasets
The framework was evaluated on several rigorous benchmarks:
- CARLA Leaderboard: The primary benchmark for urban autonomous driving.
- NoCrash Benchmark: Used to assess generalization across varying weather conditions and traffic densities.
- Expert Data: Approximately 22 hours of driving data (200,000 samples) generated by the CARLA autopilot. Notably, this data was imperfect, containing roughly 10% noisy or suboptimal choices.
- MuJoCo: Standard continuous control tasks (e.g., Humanoid, Ant, Walker2d) were used to verify the generalizability of GRI beyond the driving domain.
Key Insights and Results
Performance and Efficiency
GRIAD achieved a Driving Score of 36.79 on the CARLA leaderboard, representing a 17% improvement over the previous state-of-the-art camera-based model, World on Rails (31.37). Remarkably, GRIAD achieved these results using only three cameras, outperforming many models that rely on more expensive LiDAR sensors.
Generalization and Robustness
Ablation studies on the "NoCrash" benchmark revealed that GRI acts as a powerful regularizer. While vanilla RL agents often overfit to training weather conditions, GRI agents generalized significantly better to unseen environments by leveraging the diversity of states present in the expert data. Furthermore, the system proved robust to the 10% noise in the CARLA autopilot demonstrations, still yielding substantial improvements over pure RL.
The "Warm-up" Phase
In complex tasks like MuJoCo's HalfCheetah, the authors noted a "warm-up" period. Because expert data often represents high-velocity states that an exploration agent cannot initially reach, the agent must first learn to navigate to the regions of the state space where the expert data becomes relevant.
Conclusion
GRI offers a simple, "plug-and-play" solution for combining the strengths of IL and RL. By simply adjusting the reward values of expert transitions, it enables off-policy algorithms to learn more efficiently and generalize more effectively. While challenges such as reward inconsistency and distribution-shift-induced warm-up phases remain, GRIAD's success on the CARLA leaderboard highlights the potential of reinforced imitation for complex, vision-based robotics tasks. Future work will explore the integration of multi-modal sensor fusion to further enhance performance.
Summary updated