Explainable Action Prediction through Self-Supervision on Scene Graphs

Pawit Kochakarn, Daniele De Martini, Daniel Omeiza, Lars Kunze

Abstract

Autonomous vehicle (AV) decision-making requires robust and transparent action prediction. Traditional deep learning models often struggle with data scarcity and the inherent class imbalance of real-world driving, where critical maneuvers like overtaking are rare compared to moving straight. This paper introduces a novel framework that utilizes self-supervised learning on spatio-temporal scene graphs. By representing driving environments as structured relational data rather than raw pixels, the model achieves superior performance on imbalanced datasets while offering built-in explainability through spatial and temporal attention mechanisms.

Introduction

Predicting the future actions of a vehicle is fundamental to safe autonomous navigation. However, two primary challenges hinder current deep learning approaches: the high cost of labeled data combined with extreme class imbalance, and the "black box" nature of neural networks. In typical driving datasets, "moving straight" accounts for the vast majority of samples, leading supervised models to over-fit to common behaviors at the expense of rare, safety-critical actions.

To address these issues, the researchers from the Oxford Robotics Institute propose using scene graphs—mathematical structures where nodes represent objects (cars, pedestrians) and edges represent their relationships (distance, relative motion). This distilled representation allows models to reason about high-level semantics rather than low-level pixel data.

Methodology: The Graph Encoder Architecture

The core contribution is a specialized Graph Encoder designed to process both spatial and temporal relational information.

The Graph Encoder architecture uses a sequence of graph convolutions, pooling, and temporal attention to create a single embedding.

The pipeline consists of several key stages:

  • Spatial Processing: A Multi-Relational Graph Convolutional Network (MR-GCN) processes individual scene graphs, followed by Self-Attention Pooling (SAGPool) to identify the most salient nodes.
  • Temporal Processing: A sequence of processed graphs is fed into an LSTM with attention, allowing the model to focus on specific frames critical to predicting a future action.
  • Self-Supervised Pretraining: The model is pretrained using Graph Contrastive Learning. By creating "distorted" versions of scene graph sequences (e.g., jittering depth or dropping distant nodes) and learning to recognize them as the same scene, the model develops robust features without requiring human labels.

Experimental Setup

The framework was evaluated on the ROAD (ROad event Awareness Dataset), derived from the Oxford RobotCar Dataset. The dataset is highly imbalanced, dominated by "Move" and "Stop" actions. To enhance the scene graphs, the authors integrated proximity labels derived from depth estimation, providing the model with explicit spatial context regarding object distances to the ego-vehicle.

Results and Explainability

The self-supervised approach acts as a powerful stabilizer against class imbalance. The "Pretraining and Fine-tuning" (PF) model achieved a weighted F1-score of 0.874, significantly higher than the non-augmented supervised baseline of 0.524. Notably, the model's ability to predict rare actions like "Overtake" improved from an F1 of 0.038 to 0.708.

Beyond performance metrics, the architecture provides inherent transparency. The attention mechanisms (SAGPool and LSTM-attn) generate heatmaps that reveal the model's reasoning process.

Spatial attention scores show the model focusing on dynamic agents in the oncoming lane while ignoring a parked truck.

As shown in the visualization, the model learns to prioritize dynamic agents in the oncoming lane while discounting irrelevant objects like parked vehicles. This indicates that the model is learning meaningful driving semantics rather than just statistical correlations.

Conclusion and Future Work

This research demonstrates that self-supervision on structured scene graphs can bridge the gap between performance and explainability in AV action prediction. While the results are promising, challenges remain in perfectly capturing extremely rare classes like "Turn Left." Future work may involve integrating synthetic data for better class balancing, exploring more advanced graph layers like Relational Graph Attention (RGAT), and moving toward probabilistic modeling to better account for the stochastic nature of future driving scenarios.

Summary updated

BibTeX
@inproceedings{kochakarn2023explainable,
  author = {Pawit Kochakarn and Daniele De Martini and Daniel Omeiza and Lars Kunze},
  title = {{Explainable Action Prediction through Self-Supervision on Scene Graphs}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=I3D97wfcPo},
}

Loading related content...

Figure