RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding

Junyao Wang, Arnav Vaibhav Malawade, Junhong Zhou, Shih-Yuan Yu, Mohammad Abdullah Al FaruqueWACV 2024

TL;DR

This paper introduces RS2G, a novel framework for autonomous driving that uses a data-driven approach to extract scene graphs, representing complex interactions between road users. By learning these relationships directly from data, RS2G significantly improves risk assessment and the ability to transfer knowledge from simulations to real-world scenarios, outperforming existing methods.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) need to understand intricate interactions between road users to navigate safely. While graph learning (GL) has shown promise, existing methods rely on predefined, domain-specific rules for extracting these "scene graphs." This rigidity limits their ability to adapt to diverse, real-world scenarios and hinders the transfer of knowledge learned in simulations to actual driving conditions. Furthermore, many current deep learning approaches focus on low-level features and struggle with high-level semantic understanding, especially when dealing with complex or rare situations. This paper aims to bridge this gap by proposing a data-driven scene-graph extraction method that is more flexible, expressive, and adaptable to various driving environments.

Methods

The core innovation of RS2G lies in its data-driven scene-graph extraction. Unlike rule-based methods that predefine relationships (e.g., "car A is close to car B"), RS2G learns these relationships directly from data.

Here's a breakdown of the key components:

  • Node Encoder: Transforms object attributes (like type and location) into feature vectors.
  • Edge Encoder (Transformer-inspired): This is a crucial novelty. Instead of fixed rules, RS2G uses a Transformer-based encoder to learn the relationships between nodes. It represents these relationships as vectors, capturing latent features and enabling a more dynamic and domain-adaptive graph. This leverages the Transformer's power in capturing dependencies within input data.
  • Variational Autoencoder (VAE): Used for regularization to enhance the model's generalization capabilities. By reparameterizing relation vectors and incorporating a Kullback-Leibler (KL) divergence loss, RS2G encourages more salient and meaningful graph structures, preventing overfitting.
  • Spatial-Temporal Graph Embedding:
    • Spatial Model (MR-GCN): A multi-relational graph convolutional network processes each scene graph to capture spatial relationships between objects. It aggregates information from neighboring nodes based on different relation types.
  • Temporal Model (LSTM-attn): A Long Short-Term Memory network with an attention mechanism processes the sequence of spatial embeddings over time, capturing the temporal dynamics of the scene.
  • Risk Inference: A final multi-layer perceptron (MLP) uses the combined spatial-temporal embedding to classify the driving scenario as risky or non-risky.

Datasets

The paper uses a combination of simulated and real-world datasets for evaluation:

  • Simulated Datasets:
    • 271-carla
    • 1043-carla (These datasets focus on lane-changing scenarios.)
  • Real-World Datasets:
    • 1361-honda (clear-weather safe driving in California Bay Area)
    • 620-dash (real-world crashes and dangerous road scenarios from dash-cam footage)

The datasets are split into training (70%) and inference (30%) sets for subjective risk assessment. For transfer learning experiments, models are trained on simulation data and evaluated on real-world data.

Key Insights and Ideas

  • Data-Driven is Key: The most significant insight is that learning graph extraction rules directly from data, rather than relying on human-defined rules, dramatically improves performance. This allows the model to discover nuanced and complex relationships that might be missed by predefined rules.
  • Transformer for Relations: Using a Transformer-based edge encoder is a powerful way to capture intricate dependencies between road users. This attention mechanism allows the model to dynamically weigh the importance of different interactions.
  • VAE for Robustness: The integration of a VAE with KL divergence regularization is crucial for improving generalization. It helps the model learn more balanced and expressive scene graphs, making it less prone to overfitting and better at handling unseen scenarios.
  • Sim2Real Advantage: The data-driven approach, particularly with the Transformer, shows a significant advantage in transferring knowledge from simulations to real-world driving. This is critical for AV development, where real-world data is often scarce and expensive.

Key Results

  • Subjective Risk Assessment: RS2G (Transformer variant) achieved, on average, 21.35% higher accuracy than the SOTA DL-based model (no graph extraction) and 4.47% higher accuracy than the SOTA rule-based graph extraction method. It also showed improvements in MCC and AUC scores.
  • Robustness to Real-World Data: RS2G models exhibited significantly less performance degradation on real-world datasets compared to synthetic ones, indicating better adaptability to complex and unpredictable driving conditions.
  • Transfer Learning (Sim2Real): When transferring from simulation to a challenging real-world dataset (620-dash), RS2G (Transformer) achieved 14.68% higher accuracy than the SOTA DL-based model and 15.90% higher accuracy than the SOTA rule-based GL method.
  • Ablation Studies: The Transformer edge encoder and KL divergence regularization were shown to be critical components for achieving the best performance.

Limitations and Future Work

  • Computational Cost: While RS2G offers significant performance gains, the paper notes that DL-based approaches can be computationally expensive, which is a consideration for real-time AV systems with limited resources.
  • Edge Threshold Sensitivity: The ablation study on graph structure comparison suggests that the choice of the edge extraction threshold (γ) can impact performance, with a sweet spot around γ = 0.25 for the Transformer. Further exploration of optimal threshold selection or adaptive thresholding could be beneficial.
  • Future Directions: The authors suggest that their framework can be extended to other tasks beyond risk assessment, such as trajectory prediction and behavior forecasting. The data-driven nature of RS2G makes it a promising foundation for a wide range of autonomous driving perception tasks.

Summary updated

BibTeX
@inproceedings{wang2024rs,
  author = {Junyao Wang and Arnav Vaibhav Malawade and Junhong Zhou and Shih-Yuan Yu and Mohammad Abdullah Al Faruque},
  title = {{RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure