Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles

Victor Vaquero, Alberto Sanfeliu, Francesc Moreno-Noguer

Abstract

Optical flow estimation is a critical component of autonomous navigation, enabling tasks such as moving object detection and scene segmentation. While RGB cameras are the standard sensor for this task, they are prone to failure in adverse lighting and weather conditions. Lidar sensors offer robustness in these environments but provide only sparse 3D data, creating a resolution gap for dense motion estimation. This paper presents a novel deep learning architecture designed to "hallucinate" dense, high-resolution 2D optical flow maps from sparse Lidar scans. By employing a cross-modal teacher-student learning paradigm—using pseudo-ground truth generated from image-based models—the proposed system achieves performance competitive with traditional vision-based methods while using significantly less input data.

Introduction

Optical flow—the pattern of apparent motion of objects between consecutive frames—allows autonomous vehicles to detect dynamic obstacles and estimate time-to-collision. Traditionally, this is computed from RGB camera streams. However, cameras are "fair-weather" sensors that struggle with glare, total darkness, and heavy precipitation. Lidar sensors are robust in these conditions but present a significant challenge: they provide discrete, sparse points rather than a continuous grid of pixels. This research addresses the "sparsity gap" by developing a method to generate a dense, pixel-level motion field from scattered, low-resolution Lidar data.

Proposed Architecture

The authors propose a three-block Convolutional Neural Network (CNN) architecture specifically designed to bridge the gap between sparse Lidar scans and dense image-like flow maps:

  1. Lidar-Flow Block: This initial stage estimates motion directly in the sparse domain. It uses a contractive-expansive architecture (similar to FlowNet) to find correspondences between two consecutive Lidar scans.
  2. Lidar-to-Image Domain Transformation: This stage performs the "hallucination." The network uses multiscale filters to upscale the sparse flow. Notably, it employs asymmetric filters (e.g., $3 \times 25$) to account for the fact that Lidar data is typically much wider than it is tall, helping the network interpolate missing data horizontally.
  3. Iterative Refinement: To mitigate over-smoothing common in upscaled images, a refinement block runs five times. This acts as a lightweight alternative to a Conditional Random Field (CRF), sharpening edges and improving the definition of moving objects.
Overview of the training and inference pipeline, illustrating the teacher-student learning paradigm where an image-based model provides pseudo-ground truth for the Lidar-based network.

Data and Training

A major hurdle in this research was the lack of dense ground-truth flow labels for Lidar scans. To solve this, the authors created a "Pseudo Ground-Truth" dataset. They utilized the KITTI Tracking Benchmark, running FlowNet2 (a high-performance image-based model) on the corresponding RGB images to generate high-quality flow labels. These labels served as the target for the Lidar-based "student" network, allowing it to learn the dense motion patterns of a vision system without requiring manual human annotations.

Key Insights and Results

The core innovation of this work is the Cross-Modal Teacher-Student Learning framework. By training the Lidar network to mimic an image-based expert, the model learns to fill the gaps between sparse points effectively. Furthermore, the use of asymmetric kernels allows the network to aggregate context across the wide Lidar field of view.

Visual comparison of the hallucinated dense flow maps against input data, demonstrating the network's ability to recover object shapes and motion boundaries from sparse points.

Despite having 13 times less input data than an RGB camera, the model produced results competitive with early deep-learning image methods. The system achieved an End-Point-Error (EPE) of 5.16 on the KITTI Flow 2015 test set, performing on par with algorithms like EpicFlow and InterpoNet that use full-resolution images. Qualitatively, the hallucinated flow maps show clear, distinct shapes for vehicles and pedestrians. Additionally, the model demonstrated superior stability in occluded regions compared to many pure vision systems, as Lidar inherently captures the underlying 3D structure of the scene.

Conclusion

This work demonstrates that dense optical flow can be successfully hallucinated from sparse Lidar data, providing a robust "drop-in" replacement for cameras during adverse conditions. While foreground objects remain a challenge due to their dynamic nature and small size, the system offers a reliable path forward for all-weather autonomous navigation. Future work will focus on integrating multi-modal sensor fusion to combine the strengths of both Lidar and cameras when conditions permit.

Summary updated

BibTeX
@inproceedings{vaquero2018hallucinating,
  author = {Victor Vaquero and Alberto Sanfeliu and Francesc Moreno-Noguer},
  title = {{Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles}},
  booktitle = {ICPR},
  year = {2018},
  doi = {10.1109/icpr.2018.8545803},
}
Figure