Abstract
One way to approach end-to-end autonomous driving is to learn a policy that maps from a sensory input, such as an image frame from a front-facing camera, to a driving action, by imitating an expert driver, or a reference policy. This can be done by supervised learning, where a policy is tuned to minimize the difference between the predicted and ground-truth actions. A policy trained in this way however is known to suffer from unexpected behaviours due to the mismatch between the states reachable by the reference policy and trained policy. More advanced algorithms for imitation learning, such as DAgger, addresses this issue by iteratively collecting training examples from both reference and trained policies. These algorithms often require a large number of queries to a reference policy, which is undesirable as the reference policy is often expensive. In this paper, we propose an extension of the DAgger, called SafeDAgger, that is query-efficient and more suitable for end-to-end autonomous driving. We evaluate the proposed SafeDAgger in a car racing simulator and show that it indeed requires less queries to a reference policy. We observe a significant speed up in convergence, which we conjecture to be due to the effect of automated curriculum learning.
Introduction
End-to-end autonomous driving aims to map raw sensory inputs, like camera images, directly to control actions. While simple Supervised Learning (SL) is a straightforward approach, it often fails due to "covariate shift"—where small execution errors lead the agent into unfamiliar states, causing catastrophic failure. Dataset Aggregation (DAgger) mitigates this by iteratively involving an expert to label states visited by the agent. However, DAgger is notoriously query-expensive, requiring constant expert supervision. This is impractical for human teachers and potentially dangerous during early training stages when the student policy is highly unstable.
Methodology: The SafeDAgger Approach
SafeDAgger introduces a Safety Classifier ($c_{safe}$) to optimize the interaction between the student policy and the expert, addressing the "Nagging Student" problem where a student only needs feedback when genuinely confused.
The Safety Classifier and Strategy
The safety classifier is a secondary network that predicts the discrepancy (error) between the primary policy's action and the expert's ground truth. If the predicted error exceeds a threshold $\tau$, the state is classified as "unsafe." During data collection, the classifier acts as a gatekeeper: if a state is deemed unsafe, control is immediately handed back to the expert. This prevents crashes and allows for longer data collection runs.
Automated Curriculum Learning
SafeDAgger only adds these "unsafe" states to the training set. This creates an automated curriculum where the model first learns basic driving and then progressively masters difficult recovery maneuvers. Theoretically, the safety classifier functions similarly to a Value Function in Reinforcement Learning, predicting the "safety value" of a state under the current policy.
Experimental Setup
The authors evaluated SafeDAgger using TORCS (The Open Racing Car Simulator) across 10 tracks with realistic traffic. A rule-based AI served as the expert. To improve robustness, the primary policy was trained with auxiliary tasks, such as lane detection, alongside the main driving task.
Results and Discussion
SafeDAgger demonstrated significant improvements over standard DAgger and Supervised Learning across several key performance metrics.
Performance and Convergence
As shown in the figures above, SafeDAgger achieves "perfect" driving (completing 3 laps) significantly faster than vanilla DAgger. The "SafeDAgger-Safe" variant, which utilizes the safety classifier during evaluation, maintains zero damage throughout the training iterations.
Query Efficiency and the Virtuous Cycle
A major advantage of SafeDAgger is its query efficiency. By only querying the expert during unsafe states, the authors achieved up to a 39x reduction in expert queries compared to standard DAgger. Furthermore, as the primary policy improves, the percentage of states flagged as unsafe by $c_{safe}$ steadily decreases (Figure d). This "virtuous cycle" indicates that the student is successfully internalizing the expert's behavior, leading to increased autonomy over time.
Conclusion
SafeDAgger effectively addresses the expert query bottleneck in imitation learning while improving safety and convergence speed. By framing safety as a predictive task, the algorithm implements a practical automated curriculum, making it a robust choice for training end-to-end autonomous systems in complex, high-stakes environments.
Summary updated