Abstract
Deep learning (DL) has revolutionized autonomous driving (AD), yet the inherent opacity of these models remains a significant barrier to public trust. Current explainable autonomous driving (EAD) methods often fail to fully utilize the semantic richness and logical correlations between driving actions and their justifications. We propose Semantic-Guided Dynamic Correlation Learning (SGDCL), a framework that leverages language embeddings to guide visual attention and models dynamic interactions between categories using graph neural networks. Evaluated on the BDD-OIA and PSI benchmarks, SGDCL achieves up to a 15.3% performance improvement over state-of-the-art baselines, providing both high accuracy and interpretable visual evidence for its decisions.
Introduction
For autonomous vehicles to be safely integrated into society, they must not only act correctly but also explain why they act. Existing EAD models typically treat actions (e.g., "Stop") and explanations (e.g., "Red Light") as independent numerical labels. This approach ignores two critical pieces of information: the semantic meaning of labels that can direct visual focus, and the logical dependencies where certain scene elements necessitate specific actions. SGDCL addresses these gaps by integrating language-guided feature extraction with dynamic correlation modeling.
Proposed Method
The SGDCL framework consists of two core modules designed to enhance feature representation and logical reasoning.
Semantic-Guided Learning (SGL)
The SGL module utilizes a language encoder (Sentence-BERT) to transform textual categories into embeddings. These embeddings act as spatial pointers; through a cross-attention mechanism, the model learns to map specific textual concepts to relevant visual regions. For instance, the category "Solid Line on the Left" forces the model to attend to the left-side pixels of the input image, creating more discriminative and interpretable features.
Dynamic Correlation Learning (DCL)
To capture the interplay between different scene elements, the DCL module constructs a heterogeneous graph where nodes represent actions and explanations. Using Graph Attention Networks (GAT), the model dynamically learns how these categories influence one another in a specific context. This is supplemented by a novel correlation-based loss term ($L_{cor}$) that regularizes the model using real-world co-occurrence statistics from the training data.
Experimental Results
SGDCL was benchmarked against seven state-of-the-art models and the large vision-language model GPT-4V on the BDD-OIA and PSI datasets.
Quantitative Performance
On the BDD-OIA dataset, SGDCL demonstrated superior performance, notably achieving a 9.3% improvement in explanation mF1 over the closest competitor. When compared to GPT-4V, SGDCL proved more reliable; while GPT-4V often exhibited "overly cautious" behavior (defaulting to "Stop" unnecessarily), SGDCL provided precise actions and comprehensive explanations.
Qualitative Analysis and Explainability
Visualization of the model's internal attention maps confirms that SGDCL "looks" at the correct objects. As shown in the comparison below, SGDCL accurately identifies lane lines and obstacles that correspond to its predicted explanations, whereas baseline models like OIA often fail to localize the relevant features or miss critical explanations entirely.
Conclusion
SGDCL represents a significant step forward for explainable autonomous driving. By treating labels as semantic guides rather than mere indices and modeling the dynamic logic of driving scenes, the framework achieves state-of-the-art accuracy while remaining transparent. Future work will focus on extending this reasoning to temporal video sequences and exploring open-vocabulary explanation generation to further bridge the gap between human and machine reasoning in complex traffic environments.
Summary updated