Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering

Fouad Makiyeh, Mark Bastourous, Anass Bairouk, Wei Xiao, Mirjana Maras, Tsun-Hsuan Wang, Marc Blanchon, Ramin M. Hasani, Patrick Chareyre, Daniela RusarXiv 2024 · DOI · arXiv

Abstract

End-to-end deep learning for autonomous steering typically relies on raw RGB camera data, which can be sensitive to environmental variations. While multi-sensor fusion (e.g., LIDAR, Radar) offers robustness, it introduces significant hardware costs and complexity. This paper explores an alternative: "virtual" multimodal fusion using a single monocular camera. By extracting depth maps and optical flow from video sequences and fusing them with RGB frames, we evaluate their impact on steering performance. Our findings reveal that optical flow is a superior complementary modality compared to depth, leading to a 31% reduction in steering estimation errors.

Introduction

The challenge of autonomous driving lies in translating visual perception into precise control commands. Standard RGB-only approaches often struggle with complex lighting and road textures. While adding physical sensors like LIDAR improves performance, it increases system cost and power consumption. This study investigates whether "hidden" information within a monocular video stream—specifically motion (optical flow) and geometry (depth)—can be leveraged to achieve multi-sensor-like performance with a single camera.

Methodology

The proposed framework processes monocular RGB sequences through a multi-stage pipeline:

  1. Modality Extraction: Pre-trained models such as MiDaS and Monodepth2 generate depth maps, while the Farneback algorithm computes optical flow to capture scene dynamics.
  2. Fusion Strategies: Two approaches were tested: Early Fusion, which stacks modalities at the input level, and Hybrid Fusion, which utilizes an Attention Complementary Module (ACM) to merge features at deeper network layers.
  3. Architectural Backends: The study compared standard Convolutional Neural Networks (CNNs) against Variational Autoencoders (VAEs).
  4. Temporal Modeling: To capture the sequential nature of driving, the researchers employed Long Short-Term Memory (LSTM) networks and bio-inspired Neural Circuit Policies (NCPs).
Overview of the proposed framework, showing modality extraction and the fusion strategies within the convolution feature extractor.

Results and Key Insights

Experiments conducted on the Boston driving dataset using 10-fold cross-validation yielded several critical insights:

The Superiority of Optical Flow

Optical flow proved to be the "secret sauce" for improved steering. While depth maps share high structural similarity with RGB images (SSIM of 0.30), optical flow captures unique motion dynamics (SSIM of 0.11). This lack of redundancy allows the model to learn complementary features that RGB alone cannot provide.

Quantitative Performance Gains

The integration of optical flow led to dramatic improvements in Mean Squared Error (MSE) across all tested architectures:

  • CNN-NCP: MSE decreased from 3.68 (RGB only) to 1.64 (RGB + Optical Flow).
  • VAE-NCP: MSE decreased from 5.68 to 1.82.
  • VAE-LSTM: MSE decreased from 5.54 to 1.77.
Training and validation loss curves comparing RGB-only models with those using RGB and Optical Flow (RGB-OF) fusion.

Redundancy of Depth

Interestingly, adding depth information provided negligible benefits. The authors hypothesize that neural networks can inherently infer sufficient depth information from RGB sequences, making explicit depth maps redundant for the steering task.

Interpretability and Robustness

Using VAE-NCP architectures allowed for better model auditing. Through Automatic Latent Perturbation (ALP), the researchers found that models utilizing optical flow were more resilient to internal noise and better at identifying out-of-distribution scenarios, enhancing the safety profile of the steering system.

Conclusion

This research demonstrates that motion information is far more valuable than depth data for monocular autonomous steering. By fusing optical flow with standard RGB frames, steering estimation errors can be reduced by 31%. While computational overhead for real-time extraction remains a challenge, this approach provides a cost-effective path toward more robust autonomous driving systems using existing camera hardware. Future work will explore extending these gains to longitudinal control and complex urban maneuvers.

Summary updated

Abstract

Autonomous vehicle navigation is a key challenge in artificial intelligence, requiring robust and accurate decision-making processes. This research introduces a new end-to-end method that exploits multimodal information from a single monocular camera to improve the steering predictions for self-driving cars. Unlike conventional models that require several sensors which can be costly and complex or rely exclusively on RGB images that may not be robust enough under different conditions, our model significantly improves vehicle steering prediction performance from a single visual sensor. By focusing on the fusion of RGB imagery with depth completion information or optical flow data, we propose a comprehensive framework that integrates these modalities through both early and hybrid fusion techniques. We use three distinct neural network models to implement our approach: Convolution Neural Network - Neutral Circuit Policy (CNN-NCP) , Variational Auto Encoder - Long Short-Term Memory (VAE-LSTM) , and Neural Circuit Policy architecture VAE-NCP. By incorporating optical flow into the decision-making process, our method significantly advances autonomous navigation. Empirical results from our comparative study using Boston driving data show that our model, which integrates image and motion information, is robust and reliable. It outperforms state-of-the-art approaches that do not use optical flow, reducing the steering estimation error by 31%. This demonstrates the potential of optical flow data, combined with advanced neural network architectures (a CNN-based structure for fusing data and a Recurrence-based network for inferring a command from latent space), to enhance the performance of autonomous vehicles steering estimation.
BibTeX
@misc{makiyeh2024optical,
  author = {Fouad Makiyeh and Mark Bastourous and Anass Bairouk and Wei Xiao and Mirjana Maras and Tsun-Hsuan Wang and Marc Blanchon and Ramin M. Hasani and Patrick Chareyre and Daniela Rus},
  title = {{Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering}},
  year = {2024},
  eprint = {2409.12716},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.12716},
}
Figure