TL;DR
This paper introduces a novel end-to-end deep learning system for autonomous driving that uses a Transformer-based model to not only predict driving actions but also provide explanations for those actions. This approach aims to improve the interpretability of autonomous driving systems, fostering greater user trust and aiding in model development.
What Problem Does This Paper Address?
Deep learning (DL) models have revolutionized computer vision (CV) for autonomous driving (AD), often surpassing human performance. However, these models are typically "black boxes," lacking transparency in their decision-making processes. This lack of interpretability hinders user trust and makes it difficult to diagnose model failures, slowing down the widespread adoption of AD systems. Existing approaches often rely on pipelined systems or end-to-end models that struggle to provide clear justifications for their outputs.
Methods
The core of this paper is an end-to-end explainable autonomous driving system built upon the Transformer architecture. The proposed model consists of three main blocks:
- Feature Extractor: This block uses pre-trained Convolutional Neural Networks (CNNs) like ResNet50 or MobileNetV2 to extract visual features from input images. The spatial dimensions are then flattened to create a sequence of features.
- Multi-head Self Attention (MHSA): This is the key innovation, employing a Transformer's self-attention mechanism to perform "global soft attention." Unlike previous object-induced attention mechanisms that focused on specific cropped regions, this approach considers the entire image, mimicking human peripheral vision. It allows the model to capture long-range correlations within the image, which is crucial for understanding the broader driving context.
- Decision/Reason Generation: This block takes the fused features from the MHSA layer and uses two separate fully connected (FC) branches to classify both the driving action (e.g., move forward, turn left) and the corresponding explanation. The model is trained end-to-end by aggregating the losses from both branches.
Datasets
The models were trained and evaluated using the BDD Object Induced Actions (BDD-OIA) dataset. This dataset is an extension of the BDD-100K dataset, with each frame individually labeled for driving actions and explanations. The dataset includes actions like "move forward," "stop/slow down," "turn left," and "turn right," along with 21 classes of explanations. The paper used a training set of 16,082 images, a validation set of 2,270 images, and a test set of 4,572 images.
Key Insights and Ideas
- Global Soft Attention Mimics Peripheral Vision: The Transformer's ability to attend to global image features, rather than just specific object regions, is analogous to how humans use peripheral vision to get an overall sense of the driving environment before focusing on details. This holistic view is argued to be more critical for high-level driving decisions.
- Transformer's Long-Range Correlation Capability: The MHSA layer's strength lies in its ability to capture long-range dependencies within an image, which is a significant advantage over traditional CNNs that are limited by local receptive fields. This is vital for understanding relative spatial relationships (e.g., "left" vs. "right").
- Explainability Enhances Trust and Diagnostics: The inclusion of an "explanation head" not only provides insights into the model's reasoning but also serves as a valuable tool for debugging and validating the learned causal relationships, ultimately boosting confidence in the system.
- Soft Attention Outperforms Hard Attention: The study suggests that soft attention, which fuses information based on individual contributions, is more effective than hard attention (which selects specific regions), as it avoids information bottlenecks.
Key Results
The proposed Global Soft Attention (GSA) models demonstrated superior performance across the board compared to baseline models:
- Higher Accuracy: GSA models achieved significantly higher F1 scores for both action prediction and explanation generation, particularly for explanations. For instance, GSA (resnet) achieved an explanation F1all of 0.525, compared to Xu et al.'s (8) regional attention model which scored 0.422.
- Faster Convergence: The training curves show that the GSA models converge much faster and reach lower training losses than the baseline models.
- Computational Efficiency: The GSA model using MobileNetV2 as a feature extractor achieved comparable predictive performance to the Resnet50-based GSA model but with a significantly shorter training time (2h 53min vs. 3h 15min for 40 epochs).
- Parameter Efficiency: The GSA (mobilenet) model also uses fewer parameters (2.61M) compared to most baselines.
Limitations and Future Work
- Future Work: The authors suggest future improvements by incorporating other sensor modalities like LiDAR point clouds and vehicle-to-vehicle (V2V) connectivity. Fusing this information could provide redundancy and enhance safety.
- Limitations: While not explicitly detailed as limitations, the paper focuses on image-based driving decisions. The performance with other sensor inputs remains an open area. The dataset, while extensive, is specific to the BDD-OIA benchmark.
Summary updated