Abstract
Combining deep neural networks with reinforcement learning has shown great potential in the next-generation intelligent control. However, there are challenges in terms of safety and cost in practical applications. In this paper, we propose the Intervention Aided Reinforcement Learning (IARL) framework, which utilizes human intervened robot-environment interaction to improve the policy. We used the Unmanned Aerial Vehicle (UAV) as the test platform. We built neural networks as our policy to map sensor readings to control signals on the UAV. Our experiment scenarios cover both simulation and reality. We show that our approach substantially reduces the human intervention and improves the performance in autonomous navigation, at the same time it ensures safety and keeps training cost acceptable.
Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in simulated environments, yet its transition to real-world robotics is hindered by significant safety concerns. In simulation, an agent can experience thousands of collisions to learn optimal behavior. In reality, a single crash of an Unmanned Aerial Vehicle (UAV) can lead to hardware destruction and high costs. While high-fidelity simulations attempt to bridge the "sim-to-real" gap, discrepancies in sensor noise and physics often persist. Conversely, pure Imitation Learning (IL) methods like Behavior Cloning struggle with out-of-distribution states that the expert never encountered. This paper introduces the Intervention Aided Reinforcement Learning (IARL) framework to enable safe, efficient training in high-risk environments by leveraging human intervention as a critical learning signal.
Methodology
The IARL framework redefines the training behavior policy as a synthesis of the agent’s current policy and expert intervention. When a human expert perceives an imminent collision, they take control, providing a "safety mask."
The IARL Framework
The core of IARL lies in its dual-objective approach:
- Intervention Masking: During expert intervention, policy gradients are masked. Since the agent did not select the action, it is not directly penalized or rewarded via standard policy gradients, preventing noisy updates.
- Hybrid Loss Function: The framework employs a composite loss. One component uses RL to minimize the probability of future interventions, effectively treating the threat of collision as a penalty. The second component utilizes Imitation Learning to align the agent's policy with expert actions specifically within these "intervened zones," teaching the drone how to recover from dangerous states.
Network Architecture and Data
To address the challenges of noisy real-world sensors (depth cameras, IMUs, and supersonic sensors), the authors implemented a CNN + LSTM architecture. The LSTM module provides temporal context, allowing the model to integrate the last 2.5 seconds of flight data to filter sensor jitter and manage partially observable states.
The system was evaluated in two settings:
- Simulation: 100 scenarios in Gazebo/ROS with a simulated expert using Dijkstra’s algorithm and Model Predictive Control.
- Real-World: An indoor environment with foam obstacles using an X-type quadrotor powered by an Nvidia Jetson TX2. Real-world training was initialized with 100,000 frames of expert pre-training.
Results and Discussion
The IARL framework demonstrated significant improvements in both safety and performance metrics compared to traditional baselines.
Performance Comparison
In simulation tests without expert assistance, IARL achieved a 0% average collision rate, whereas standard RL and DAgger suffered from 22.5% and 24% collision rates, respectively. The reward curves indicate that IARL converges faster and reaches higher performance levels than pure RL or imitation-based methods.
Key Insights
- Safety as a Teaching Signal: By learning from the threat of collision (intervention) rather than the collision itself, the agent develops a conservative value function that avoids high-risk areas.
- Recovery Learning: Standard RL often fails in states it has been "saved" from because it never experiences the subsequent recovery. IARL’s imitation loss in intervened zones ensures the agent learns safe recovery maneuvers.
- Temporal Stability: The recurrent CNN+LSTM structure significantly outperformed single-frame and frame-stacking models, achieving a Mean Square Error (MSE) of 0.0044, proving that temporal memory is essential for stable autonomous flight.
Conclusion
IARL offers a practical path toward deploying RL in high-stakes real-world applications by treating human intervention as a valuable data source rather than just a safety net. While the framework currently requires an expert "in the loop" and faces challenges from system latency (0.3s–0.4s), it drastically reduces the hardware risks associated with autonomous learning. Future research will explore applying IARL to other domains, such as autonomous driving, where safety-critical interventions are already a standard performance metric.
Summary updated