Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction

Sarah Friday, Yunzi Shi, Yaswanth Cherivirala, Vishwanath Saragadam, Adithya PediredlaCVPR 2024 · DOI

Introduction

Modern "flash" Lidar systems, commonly found in high-end smartphones and autonomous robotics, primarily utilize Amplitude Modulated Continuous Wave (AMCW) Time-of-Flight (ToF) sensors. While effective, these sensors typically rely on a "four-tap" approach, requiring four sequential image captures (quadrature measurements) with varying phase shifts to calculate a single depth map. This multi-frame requirement introduces two significant bottlenecks: motion artifacts, where movement between frames leads to physical inaccuracies, and high data bandwidth requirements that limit the maximum achievable frame rate for real-time applications like high-speed robotics or augmented reality.

Snapshot Lidar addresses these challenges by enabling the simultaneous capture of both scene intensity (amplitude) and depth (phase) in a single frame. By achieving a 4x boost in temporal resolution with minimal hardware modifications, this approach offers a robust solution for dynamic environments.

Methodology: Emulating Off-Axis Holography

The core innovation of this work is the emulation of off-axis holography within a digital ToF sensor. In traditional optical holography, a tilted reference beam interferes with a signal beam to create a spatial carrier frequency. This frequency separates the desired image from its "twin" in the Fourier (frequency) domain, allowing for clean reconstruction.

Instead of a physical tilted laser beam, the researchers leveraged the rolling shutter effect inherent in many CMOS sensors. As the sensor reads the image row-by-row, the system linearly varies the phase of the light source. This process encodes depth information into a specific high-frequency band of the single captured image, effectively creating a "ToF hologram."

The rolling shutter mechanism used to emulate off-axis holography by varying phase shifts across rows.

The Reconstruction Pipeline

To recover depth from the captured snapshot, the system follows a four-step Fourier-based pipeline:

  1. Capture: A single "ToF hologram" is recorded using the phase-shifting rolling shutter.
  2. Fourier Transform: The image is converted into the frequency domain.
  3. Filter and Shift: The hologram signal, containing both amplitude and phase, is isolated and shifted back to the center of the spectrum.
  4. Inverse Fourier Transform: The signal is converted back to the spatial domain to recover a complex-valued image. The magnitude of this image represents scene brightness, while the angle represents depth.

To prevent aliasing—where high-frequency noise interferes with depth data—the researchers introduced a controlled optical prefiltering (slight blurring) using a defocused or cylindrical lens.

Experimental Setup and Validation

The system was validated using a lab prototype featuring a Melexis 75027 ToF sensor and a galvanic mirror system to simulate high-speed rolling shutter effects. The researchers evaluated the system across diverse indoor scenes, including diffuse, specular, and refractive materials, to ensure robustness across different physical properties.

Key Results and Insights

The Snapshot Lidar approach demonstrates several critical advantages over traditional four-tap systems:

  • 4x Temporal Speedup: Depth maps are reconstructed from a single measurement, quadrupling the effective frame rate.
  • Comparable Accuracy: Quantitative analysis showed a Signal-to-Noise Ratio (SNR) of 30-33 dB, matching the accuracy of traditional multi-frame methods.
  • Inherent Motion Resilience: By requiring only one frame, the system is naturally immune to the inter-frame motion artifacts that typically plague AMCW sensors.
  • Bandwidth Efficiency: Reducing the required frames per depth map significantly lowers the data throughput requirements for the camera's processing unit.
A high-quality depth map reconstructed from a single snapshot using the proposed method.

One of the most significant insights of this work is the favorable trade-off between spatial and temporal resolution. While the required optical blurring results in some loss of fine spatial detail, the gain in temporal resolution is invaluable for high-speed applications where motion blur is a primary failure mode.

Limitations and Future Directions

Despite its advantages, the current implementation has limitations. The reliance on optical prefiltering means very fine spatial details may be lost. Additionally, the method requires precise control over the light source's phase synchronization with the sensor's row-readout, a feature not yet standard in all consumer-grade mobile sensors.

Future research may explore the use of machine learning, such as Deep Image Priors, to replace manual filtering steps. This could potentially recover lost spatial resolution while maintaining the high-speed, single-shot benefits of the Snapshot Lidar architecture.

Summary updated

BibTeX
@inproceedings{friday2024snapshot,
  author = {Sarah Friday and Yunzi Shi and Yaswanth Cherivirala and Vishwanath Saragadam and Adithya Pediredla},
  title = {{Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02381},
}
Figure