Enhancing Steering Estimation with Semantic-Aware GNNs

Fouad Makiyeh, Huy-Dung Nguyen, Patrick Chareyre, Ramin M. Hasani, Marc Blanchon, Daniela RusarXiv 2025 · DOI · arXiv

Abstract

Traditional autonomous driving systems often struggle with spatial awareness when relying solely on 2D image-based models. This paper presents a novel hybrid architecture that leverages Graph Neural Networks (GNNs) and "pseudo-3D" point clouds reconstructed from monocular images. By incorporating semantic-aware graph pruning, the proposed model achieves a 71% improvement in steering angle estimation over traditional 2D baselines. Our findings suggest that dense pseudo-3D data can outperform sparse LiDAR for road geometry tasks, providing a cost-effective path toward high-precision autonomous navigation.

Introduction

Estimating accurate steering angles is a fundamental challenge in autonomous driving. While 2D convolutional neural networks (CNNs) are widely used due to their efficiency, they lack the explicit depth perception required for complex maneuvers or occluded environments. High-end sensors like LiDAR provide this depth but are expensive and often yield sparse data at long ranges.

Researchers from MIT and Capgemini Engineering address this gap by proposing a system that "lifts" 2D visual data into a structured 3D graph representation. This approach combines the cost-effectiveness of monocular cameras with the structural richness of 3D spatial reasoning.

Methodology

The proposed pipeline transitions from raw 2D input to a temporal 3D feature analysis. The architecture evolves from standard 2D CNN-RNN models to a sophisticated hybrid that uses semantic and depth information to build a 3D point cloud.

Comparison of architectural approaches: (I) 2D baseline, (II) LiDAR-based 3D, and (III) the proposed monocular pseudo-3D pipeline.

Pseudo-3D Reconstruction and Semantic Pruning

The core of the method involves a multi-step data transformation:

  1. Feature Extraction: A pretrained unified encoder-decoder estimates depth and semantic labels from a single RGB image.
  2. Point Cloud Generation: These outputs are used to project pixels into a 3D "pseudo-point cloud."
  3. Graph Construction: To manage computational complexity, the system employs semantic-aware pruning. While connections between points of the same class (e.g., road-to-road) are preserved, 80% of inter-class edges (e.g., road-to-pedestrian) are removed. This focuses the GNN on critical spatial relationships and acts as a regularizer.
Visualization of the reconstruction pipeline: RGB input, depth estimation, semantic segmentation, and the resulting 3D point cloud.

Temporal Modeling

The spatial features extracted by the GNN are processed through a recurrent neural network (RNN). Specifically, the authors utilize Neural Circuit Policies (NCPs)—biologically inspired "liquid" networks—to capture the temporal dynamics of vehicle motion.

Experimental Results

The model was evaluated using the KITTI raw dataset, specifically focusing on challenging "city" and "residential" sequences.

Performance Gains

The final model, Sa-GNN-NCP, achieved a Mean Squared Error (MSE) of 0.0771, representing a 71% improvement over the 2D-only baseline. Notably, this model also outperformed GNNs trained on raw LiDAR data (MSE 0.132). This suggests that the density of image-derived pseudo-3D maps provides a more coherent representation of road geometry than sparse LiDAR returns.

Key Insights

  • GNN Superiority: GNNs outperformed PointNet++ for steering tasks because they explicitly model point proximity, leading to smoother feature extraction.
  • Pruning Efficacy: Reducing inter-class connectivity by 80% improved accuracy by filtering out irrelevant spatial noise.
  • Trajectory Stability: Integrating the predicted steering angles showed that the Sa-GNN-NCP model maintained a stable path alignment with ground truth, whereas other 3D models exhibited significant drift.

Conclusion and Future Work

This research demonstrates that 3D spatial reasoning is essential for precise steering estimation and can be achieved without expensive LiDAR hardware. By combining monocular depth estimation with semantic-aware GNNs, the system achieves state-of-the-art performance. Future work will focus on reducing the computational overhead of 3D graph processing for real-time edge deployment and integrating high-level decision-making logic for complex driving scenarios.

Summary updated

Abstract

Steering estimation is a critical task in autonomous driving, traditionally relying on 2D image-based models. In this work, we explore the advantages of incorporating 3D spatial information through hybrid architectures that combine 3D neural network models with recurrent neural networks (RNNs) for temporal modeling, using LiDAR-based point clouds as input. We systematically evaluate four hybrid 3D models, all of which outperform the 2D-only baseline, with the Graph Neural Network (GNN) - RNN model yielding the best results. To reduce reliance on LiDAR, we leverage a pretrained unified model to estimate depth from monocular images, reconstructing pseudo-3D point clouds. We then adapt the GNN-RNN model, originally designed for LiDAR-based point clouds, to work with these pseudo-3D representations, achieving comparable or even superior performance compared to the LiDAR-based model. Additionally, the unified model provides semantic labels for each point, enabling a more structured scene representation. To further optimize graph construction, we introduce an efficient connectivity strategy where connections are predominantly formed between points of the same semantic class, with only 20\% of inter-class connections retained. This targeted approach reduces graph complexity and computational cost while preserving critical spatial relationships. Finally, we validate our approach on the KITTI dataset, achieving a 71% improvement over 2D-only models. Our findings highlight the advantages of 3D spatial information and efficient graph construction for steering estimation, while maintaining the cost-effectiveness of monocular images and avoiding the expense of LiDAR-based systems.
BibTeX
@misc{makiyeh2025enhancing,
  author = {Fouad Makiyeh and Huy-Dung Nguyen and Patrick Chareyre and Ramin M. Hasani and Marc Blanchon and Daniela Rus},
  title = {{Enhancing Steering Estimation with Semantic-Aware GNNs}},
  year = {2025},
  eprint = {2503.17153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17153},
}
Figure