Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight

Jiaxu Xing, Angel Romero, Leonard Bauersfeld, Davide Scaramuzza

Introduction

Achieving agile flight in autonomous drones—maneuvering through complex environments at high speeds—remains one of the most significant challenges in robotics. While human pilots can navigate intricate race courses using only visual feedback, robotic systems often rely on expensive, high-precision sensors or perfect state information (like GPS or motion capture) to achieve similar feats.

Developing a vision-based system for agile flight presents a "Goldilocks" problem. Reinforcement Learning (RL) is capable of discovering optimal, high-performance behaviors through trial and error, but it is notoriously sample-inefficient when dealing with high-dimensional visual data. Conversely, Imitation Learning (IL) is efficient at teaching a model to mimic an expert, but it is limited by the expert's performance ceiling and suffers from "covariate shift," where small errors accumulate into catastrophic failures. This paper introduces a three-stage framework that bootstraps RL with IL to enable drones to fly through complex race courses using only visual input, reaching speeds comparable to world-class human pilots.

The Teacher-Student-Refiner Framework

To bridge the gap between the efficiency of IL and the performance potential of RL, the researchers developed a three-phase pipeline designed to transition a policy from privileged state-based control to robust vision-based autonomy.

Stage I: The State-Based Teacher

The process begins by training an "expert" policy using Reinforcement Learning. This teacher policy is granted "privileged information," meaning it has access to the drone's exact position, velocity, and orientation. Because it does not have to interpret noisy visual data, the teacher can quickly learn the optimal racing line and control logic required for high-speed navigation.

Stage II: The Vision-Based Student

The knowledge from the teacher is then distilled into a student policy using DAgger (Dataset Aggregation). Unlike the teacher, the student is restricted to visual inputs—either raw RGB images (pixel-based) or 2D gate corners (corner-based). By mimicking the teacher's actions in various scenarios, the student develops a solid baseline policy that understands how to navigate the course, though it remains limited by the teacher's original performance.

Stage III: Adaptive RL Fine-Tuning

The final and most critical stage involves putting the student policy back into an RL environment for fine-tuning. This allows the drone to push past the teacher's limits. To ensure stability during this phase, the authors implemented two key innovations:

  1. Asymmetric Critic: During training, the "Critic" (the component that evaluates the quality of actions) has access to the full robot state, while the "Actor" (the component that actually flies) sees only the camera feed. This "cheating" during training provides the actor with high-quality feedback, accelerating the learning process.
  2. Adaptive Updates: To solve the "stability-plasticity dilemma"—where a model might forget how to fly while trying to learn new, faster maneuvers—the researchers developed an algorithm that dynamically adjusts the learning rate and clipping range based on the drone's real-time performance.

Experimental Results and Performance

The framework was evaluated on three distinct racing tracks in a high-fidelity simulator: SplitS (vertical maneuvers), Figure 8 (high-speed loops), and Kidney (tight, complex turns). The results demonstrated a significant leap over traditional methods:

  • Overcoming RL Limitations: Standard RL trained from scratch using visual inputs failed completely, achieving a 0% success rate across all tracks. In contrast, the bootstrapped approach achieved success rates of up to 85%.
  • Surpassing the Expert: The fine-tuning phase successfully pushed the policy beyond the imitation baseline. On the SplitS track, the fine-tuned policy clocked a lap time of 7.77 seconds, significantly faster than the 8.30 seconds achieved by the IL-only student.
  • Human-Level Agility: When thrust limits were increased, the vision-based policy achieved lap times within 0.05 seconds of state-of-the-art controllers that use perfect state information—controllers that have previously outperformed world-champion human pilots.
  • Robustness: The system proved resilient to perceptual disturbances, such as random frame blackouts or simulated wind-induced positional jumps, suggesting a high degree of internal stability.

Conclusion and Future Work

This research demonstrates that vision-only agile flight is not only possible but can reach elite performance levels by effectively combining imitation and reinforcement learning. By using a history of visual observations (via Temporal Convolutional Networks), the drone learns to implicitly understand its own dynamics without needing dedicated IMU data or metric state estimation.

While the results are a major step forward, the authors note that the current perception modules are primarily tested in controlled, simulated environments. Future work will focus on improving "sim-to-real" transfer, enabling these drones to generalize their visual understanding to the unpredictable lighting and background clutter of outdoor, real-world environments.

Summary updated

Abstract

Learning visuomotor policies for agile quadrotor flight presents significant difficulties, primarily from inefficient policy exploration caused by high-dimensional visual inputs and the need for precise and low-latency control. To address these challenges, we propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) in the task of vision-based autonomous drone racing. While RL provides a framework for learning high-performance controllers through trial and error, it faces challenges with sample efficiency and computational demands due to the high dimensionality of visual inputs. Conversely, IL efficiently learns from visual expert demonstrations, but it remains limited by the expert's performance and state distribution. To overcome these limitations, our policy learning framework integrates the strengths of both approaches. Our framework contains three phases: training a teacher policy using RL with privileged state information, distilling it into a student policy via IL, and adaptive fine-tuning via RL. Testing in both simulated and real-world scenarios shows our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance, successfully navigating a quadrotor through a race course using only visual information.
BibTeX
@inproceedings{xing2024bootstrapping,
  author = {Jiaxu Xing and Angel Romero and Leonard Bauersfeld and Davide Scaramuzza},
  title = {{Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=bt0PX0e4rE},
}
Figure