Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor

Masaya Horio, Yu Feng, Tomoya Kokado, Taishi Takasawa, Keita Yasutomi, Shoji Kawahito, Takashi Komuro, Hajime Nagahara, Keiichiro Kagawa

TL;DR

This paper introduces a novel multi-tap macro-pixel CMOS image sensor designed to overcome multi-path interference in Time-of-Flight (ToF) depth imaging. By acquiring multiple temporally compressed images with unique shutter patterns, the sensor can effectively separate and resolve multiple reflections, significantly improving depth accuracy in challenging scenarios.

What Problem Does This Paper Address?

Time-of-Flight (ToF) depth imaging is a powerful technique for capturing 3D information, widely used in applications like autonomous driving and robotics. However, a significant challenge in ToF systems, particularly indirect ToF (iToF), is "multi-path interference." This occurs when light emitted from the sensor bounces off multiple surfaces before returning, leading to erroneous depth measurements. Imagine trying to measure the distance to a wall, but the light also bounces off a nearby glass table – the sensor might get confused by the combined reflections. This paper specifically tackles multi-path interference caused by surface reflections from transparent objects, which is a common and difficult problem to solve with conventional ToF sensors.

Methods

The core innovation lies in the multi-tap macro-pixel computational CMOS image sensor. Unlike traditional sensors that capture a single image per pixel, this sensor's macro-pixels are composed of four subpixels, each with four "taps." These taps can be independently controlled with different temporal shutter patterns. This allows the sensor to simultaneously capture 16 different "compressed" images of the scene in a single shot, each corresponding to a unique shutter sequence.

The depth estimation process is then a two-stage approach:

  1. Coarse Estimation: A compressive sensing solver (specifically, TVAL3) is used to reconstruct the scene's response from the 16 compressed images and their corresponding shutter patterns. This step aims to identify the primary depth components.
  2. Refinement: A nonlinear search algorithm (fminsearchbnd) is employed to refine the estimated depths and amplitudes of the identified reflections, achieving sub-clock accuracy.

The use of multi-frequency shutters and sub-clock shifting in the shutter patterns is crucial for disentangling the different light paths. Multi-frequency shutters allow for capturing information at different temporal resolutions, while sub-clock shifting helps to improve depth resolution without sacrificing the overall depth range.

Datasets

Not discussed in the paper. The evaluation relies on simulations and experimental measurements using a controlled setup.

Key Insights and Ideas

  • Compressive Sensing for ToF: The paper demonstrates the power of compressive sensing principles in the context of ToF imaging. By strategically undersampling the temporal information (via shutter patterns), it's possible to reconstruct a richer temporal signal than directly captured, enabling multi-path separation.
  • Macro-Pixel Architecture as a Solution: Instead of complex multi-aperture systems, the proposed macro-pixel design with multi-tap subpixels offers a more integrated and potentially simpler solution for capturing the necessary temporal information for multi-path separation.
  • Two-Stage Depth Estimation is Effective: The combination of a compressive sensing solver for initial separation and a nonlinear refinement step proves effective in accurately estimating depths even in the presence of multiple reflections.
  • Sub-Clock Shifting for Resolution: The clever use of sub-clock shifting in the shutter patterns is a key technique for enhancing depth resolution without compromising the sensor's ability to capture distant objects.

Key Results

  • Simulation: In simulations, the proposed method achieved relative mean errors of less than 1.2% for both objective and interference depths, even with varying photon counts and interference amplitudes. The method demonstrated robustness across a range of simulated depths (1-32m).
  • Experiment: In a real-world experiment, the system successfully separated two reflections from an acrylic plate (1-2m) and a mirror (9.3m). The absolute mean error of the separated depths compared to single-path measurements was less than 0.2m.
  • Processing Time: While the depth refinement stage is computationally intensive (occupying 99.3% of the processing time in the experiment), the overall approach is shown to be viable for separating multi-path components.

Limitations and Future Work

  • Processing Speed: The current implementation's processing time, particularly the nonlinear optimization step, is too slow for real-time applications. The authors suggest exploring algorithms suitable for real-time processing as a future direction.
  • Sensor Limitations: The prototype sensor exhibits limitations such as low photosensitivity, large pixel size, and slow operating frequency, which impact performance.
  • Resolvable Paths: The number of resolvable paths is a combinatorial problem, and while the current method handles dual-path scenarios well, extending it to more complex multi-path situations might require further algorithmic improvements or more sophisticated shutter patterns.
  • Sub-Surface Scattering: The current model and method primarily focus on surface reflections. Addressing sub-surface scattering, which decays much faster, would require different techniques.
  • Point Spread Function: The assumption of uniform illumination across a macro-pixel might need to be refined by considering the point spread function for subpixel resolution.

Summary updated

Abstract

Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.
BibTeX
@inproceedings{horio2022resolving,
  author = {Masaya Horio and Yu Feng and Tomoya Kokado and Taishi Takasawa and Keita Yasutomi and Shoji Kawahito and Takashi Komuro and Hajime Nagahara and Keiichiro Kagawa},
  title = {{Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=NSyaDLXgB7z},
}
Figure