TL;DR
This paper introduces a novel multi-tap macro-pixel CMOS image sensor designed to overcome multi-path interference in Time-of-Flight (ToF) depth imaging. By acquiring multiple temporally compressed images with unique shutter patterns, the sensor can effectively separate and resolve multiple reflections, significantly improving depth accuracy in challenging scenarios.
What Problem Does This Paper Address?
Time-of-Flight (ToF) depth imaging is a powerful technique for capturing 3D information, widely used in applications like autonomous driving and robotics. However, a significant challenge in ToF systems, particularly indirect ToF (iToF), is "multi-path interference." This occurs when light emitted from the sensor bounces off multiple surfaces before returning, leading to erroneous depth measurements. Imagine trying to measure the distance to a wall, but the light also bounces off a nearby glass table – the sensor might get confused by the combined reflections. This paper specifically tackles multi-path interference caused by surface reflections from transparent objects, which is a common and difficult problem to solve with conventional ToF sensors.
Methods
The core innovation lies in the multi-tap macro-pixel computational CMOS image sensor. Unlike traditional sensors that capture a single image per pixel, this sensor's macro-pixels are composed of four subpixels, each with four "taps." These taps can be independently controlled with different temporal shutter patterns. This allows the sensor to simultaneously capture 16 different "compressed" images of the scene in a single shot, each corresponding to a unique shutter sequence.
The depth estimation process is then a two-stage approach:
- Coarse Estimation: A compressive sensing solver (specifically, TVAL3) is used to reconstruct the scene's response from the 16 compressed images and their corresponding shutter patterns. This step aims to identify the primary depth components.
- Refinement: A nonlinear search algorithm (fminsearchbnd) is employed to refine the estimated depths and amplitudes of the identified reflections, achieving sub-clock accuracy.
The use of multi-frequency shutters and sub-clock shifting in the shutter patterns is crucial for disentangling the different light paths. Multi-frequency shutters allow for capturing information at different temporal resolutions, while sub-clock shifting helps to improve depth resolution without sacrificing the overall depth range.
Datasets
Not discussed in the paper. The evaluation relies on simulations and experimental measurements using a controlled setup.
Key Insights and Ideas
- Compressive Sensing for ToF: The paper demonstrates the power of compressive sensing principles in the context of ToF imaging. By strategically undersampling the temporal information (via shutter patterns), it's possible to reconstruct a richer temporal signal than directly captured, enabling multi-path separation.
- Macro-Pixel Architecture as a Solution: Instead of complex multi-aperture systems, the proposed macro-pixel design with multi-tap subpixels offers a more integrated and potentially simpler solution for capturing the necessary temporal information for multi-path separation.
- Two-Stage Depth Estimation is Effective: The combination of a compressive sensing solver for initial separation and a nonlinear refinement step proves effective in accurately estimating depths even in the presence of multiple reflections.
- Sub-Clock Shifting for Resolution: The clever use of sub-clock shifting in the shutter patterns is a key technique for enhancing depth resolution without compromising the sensor's ability to capture distant objects.
Key Results
- Simulation: In simulations, the proposed method achieved relative mean errors of less than 1.2% for both objective and interference depths, even with varying photon counts and interference amplitudes. The method demonstrated robustness across a range of simulated depths (1-32m).
- Experiment: In a real-world experiment, the system successfully separated two reflections from an acrylic plate (1-2m) and a mirror (9.3m). The absolute mean error of the separated depths compared to single-path measurements was less than 0.2m.
- Processing Time: While the depth refinement stage is computationally intensive (occupying 99.3% of the processing time in the experiment), the overall approach is shown to be viable for separating multi-path components.
Limitations and Future Work
- Processing Speed: The current implementation's processing time, particularly the nonlinear optimization step, is too slow for real-time applications. The authors suggest exploring algorithms suitable for real-time processing as a future direction.
- Sensor Limitations: The prototype sensor exhibits limitations such as low photosensitivity, large pixel size, and slow operating frequency, which impact performance.
- Resolvable Paths: The number of resolvable paths is a combinatorial problem, and while the current method handles dual-path scenarios well, extending it to more complex multi-path situations might require further algorithmic improvements or more sophisticated shutter patterns.
- Sub-Surface Scattering: The current model and method primarily focus on surface reflections. Addressing sub-surface scattering, which decays much faster, would require different techniques.
- Point Spread Function: The assumption of uniform illumination across a macro-pixel might need to be refined by considering the point spread function for subpixel resolution.
Summary updated