SGDCL: Semantic-Guided Dynamic Correlation Learning for Explainable Autonomous Driving

Chengtai Cao, Xinhong Chen, Jianping Wang, Qun Song, Rui Tan, Yung-Hui LiIJCAI 2024 · OpenReview

Abstract

Deep learning (DL) has revolutionized autonomous driving (AD), yet the inherent opacity of these models remains a significant barrier to public trust. Current explainable autonomous driving (EAD) methods often fail to fully utilize the semantic richness and logical correlations between driving actions and their justifications. We propose Semantic-Guided Dynamic Correlation Learning (SGDCL), a framework that leverages language embeddings to guide visual attention and models dynamic interactions between categories using graph neural networks. Evaluated on the BDD-OIA and PSI benchmarks, SGDCL achieves up to a 15.3% performance improvement over state-of-the-art baselines, providing both high accuracy and interpretable visual evidence for its decisions.

Introduction

For autonomous vehicles to be safely integrated into society, they must not only act correctly but also explain why they act. Existing EAD models typically treat actions (e.g., "Stop") and explanations (e.g., "Red Light") as independent numerical labels. This approach ignores two critical pieces of information: the semantic meaning of labels that can direct visual focus, and the logical dependencies where certain scene elements necessitate specific actions. SGDCL addresses these gaps by integrating language-guided feature extraction with dynamic correlation modeling.

Proposed Method

The SGDCL framework consists of two core modules designed to enhance feature representation and logical reasoning.

Semantic-Guided Learning (SGL)

The SGL module utilizes a language encoder (Sentence-BERT) to transform textual categories into embeddings. These embeddings act as spatial pointers; through a cross-attention mechanism, the model learns to map specific textual concepts to relevant visual regions. For instance, the category "Solid Line on the Left" forces the model to attend to the left-side pixels of the input image, creating more discriminative and interpretable features.

Dynamic Correlation Learning (DCL)

To capture the interplay between different scene elements, the DCL module constructs a heterogeneous graph where nodes represent actions and explanations. Using Graph Attention Networks (GAT), the model dynamically learns how these categories influence one another in a specific context. This is supplemented by a novel correlation-based loss term ($L_{cor}$) that regularizes the model using real-world co-occurrence statistics from the training data.

The SGDCL architecture, featuring Semantic-Guided Learning and Dynamic Correlation Learning modules.

Experimental Results

SGDCL was benchmarked against seven state-of-the-art models and the large vision-language model GPT-4V on the BDD-OIA and PSI datasets.

Quantitative Performance

On the BDD-OIA dataset, SGDCL demonstrated superior performance, notably achieving a 9.3% improvement in explanation mF1 over the closest competitor. When compared to GPT-4V, SGDCL proved more reliable; while GPT-4V often exhibited "overly cautious" behavior (defaulting to "Stop" unnecessarily), SGDCL provided precise actions and comprehensive explanations.

Qualitative Analysis and Explainability

Visualization of the model's internal attention maps confirms that SGDCL "looks" at the correct objects. As shown in the comparison below, SGDCL accurately identifies lane lines and obstacles that correspond to its predicted explanations, whereas baseline models like OIA often fail to localize the relevant features or miss critical explanations entirely.

Qualitative comparison between the OIA baseline and SGDCL, showing heatmaps and predicted logical graphs.

Conclusion

SGDCL represents a significant step forward for explainable autonomous driving. By treating labels as semantic guides rather than mere indices and modeling the dynamic logic of driving scenes, the framework achieves state-of-the-art accuracy while remaining transparent. Future work will focus on extending this reasoning to temporal video sequences and exploring open-vocabulary explanation generation to further bridge the gap between human and machine reasoning in complex traffic environments.

Summary updated

Abstract

By learning expressive representations, deep learning (DL) has revolutionized autonomous driving (AD). Despite significant advancements, the inherent opacity of DL models engenders public distrust, impeding their widespread adoption. For explainable autonomous driving, current studies primarily concentrate on extracting features from input scenes to predict driving actions and their corresponding explanations. However, these methods underutilize semantics and correlation information within actions and explanations (collectively called categories in this work), leading to suboptimal performance. To address this issue, we propose Semantic-Guided Dynamic Correlation Learning (SGDCL), a novel approach that effectively exploits semantic richness and dynamic interactions intrinsic to categories. SGDCL employs a semantic-guided learning module to obtain category-specific representations and a dynamic correlation learning module to adaptively capture intricate correlations among categories. Additionally, we introduce an innovative loss term to leverage fine-grained co-occurrence statistics of categories for refined regularization. We extensively evaluate SGDCL on two well-established benchmarks, demonstrating its superiority over seven state-of-the-art baselines and a large vision-language model. SGDCL significantly promotes explainable autonomous driving with up to 15.3% performance improvement and interpretable attention scores, bolstering public trust in AD.
BibTeX
@inproceedings{cao2024sgdcl,
  author = {Chengtai Cao and Xinhong Chen and Jianping Wang and Qun Song and Rui Tan and Yung-Hui Li},
  title = {{SGDCL: Semantic-Guided Dynamic Correlation Learning for Explainable Autonomous Driving}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=iKVMJ5gH7o},
}
Figure