TL;DR
This paper introduces a novel "pseudo-dToF" CMOS image sensor that significantly improves depth quality by combining the high spatial resolution of indirect ToF (iToF) with the accuracy and multipath interference (MPI) robustness of direct ToF (dToF). By employing time-compressive sensing and novel techniques like sub-time windowing and oversampling with quadratic fitting, the sensor achieves centimeter-level depth precision and effectively separates objects even in challenging, reflective environments.
What Problem Does This Paper Address?
Traditional Time-of-Flight (ToF) depth imaging sensors face a fundamental trade-off:
- Direct ToF (dToF): Offers high accuracy and robustness to multipath interference (MPI) – where light reflects off multiple surfaces before reaching the sensor, causing depth errors. However, dToF sensors typically have lower spatial resolution due to the complex circuitry (like time-to-digital converters) required for each pixel.
- Indirect ToF (iToF): Achieves high spatial resolution with simpler pixel designs. However, iToF sensors are more susceptible to MPI, leading to inaccurate depth measurements, and can suffer from motion artifacts in dynamic scenes.
This leaves a gap for sensors that can provide both high spatial resolution and robust, accurate depth measurements, especially in environments with significant interreflections and scattering, such as those found in robotics and autonomous driving.
Methods
The core innovation is the pseudo-direct Time-of-Flight (pseudo-dToF) approach. It leverages an iToF architecture with high-speed charge modulators to achieve high spatial resolution. The key techniques employed are:
- Time-Compressive Sensing (TCS): Instead of directly measuring the time-of-flight, the sensor captures temporally compressed signals in the charge domain. This is achieved by applying random exposure codes (binary "1"s and "0"s) to shutter controllers within each pixel's sub-pixels. This allows the entire light waveform to be reconstructed from a single shot, similar to dToF, but with the iToF sensor's inherent high spatial resolution.
- Image Reconstruction: A compressive sensing algorithm, specifically using L1-norm minimization (total variation), is used to reconstruct the original, high-resolution light waveforms from the temporally compressed images. This process is mathematically described by $y = Ax$, where $y$ is the compressed measurement, $x$ is the original signal, and $A$ is the spatio-temporal observation matrix derived from exposure codes and sensor characteristics.
- Depth Calculation: Once the light waveforms are reconstructed, depth is calculated by detecting the peak arrival time of the waveform, analogous to traditional dToF.
- Depth Quality Improvement Techniques: To overcome limitations in depth precision, two methods were introduced without hardware changes:
- Sub-Time Window: The on-chip phase-locked loop (PLL) clock frequency was doubled from 303 MHz to 607 MHz. This effectively halves the minimum time window duration, improving temporal resolution.
- Oversampling and Quadratic Fitting: The sensor response (IRF) was oversampled (10x sampling per bit of exposure code). This generates more data points for reconstruction. Subsequently, quadratic fitting is applied to the five data points around the detected peak in the reconstructed waveform to precisely estimate the peak's true position, leading to sub-bin depth resolution.
Datasets
Not discussed in the paper. The experiments were conducted using a custom-built prototype sensor and controlled laboratory setups with specific objects and lighting conditions.
Key Insights and Ideas
- Bridging the dToF/iToF Divide: The pseudo-dToF method successfully merges the strengths of both dToF (accuracy, MPI robustness) and iToF (high spatial resolution) into a single sensor architecture.
- Single-Shot Waveform Reconstruction: The use of time-compressive sensing allows for the reconstruction of the entire light waveform in a single shot, eliminating motion artifacts that plague traditional iToF methods.
- Hardware Agnostic Depth Improvement: The sub-time window and oversampling/fitting techniques demonstrate that significant improvements in depth resolution and precision can be achieved through software and signal processing alone, without requiring new sensor hardware.
- MPI Mitigation: The experimental results show a remarkable ability to separate objects at different depths (e.g., 0.65m and 0.40m) even when MPI is intentionally introduced, highlighting the robustness of the approach.
- Centimeter-Level Precision: The combination of techniques, particularly 10x oversampling and quadratic fitting, pushed the depth precision down to approximately 1 cm.
Key Results
- MPI Separation: Successfully separated two reflections 40 cm apart under MPI conditions.
- Depth Precision: Achieved a depth precision of around 1 cm with 10x oversampling and quadratic fitting.
- Temporal Resolution: Improved temporal resolution to 1.65 ns (equivalent to 0.25 m depth resolution) using the sub-time window technique.
- Robustness: Demonstrated immunity to MPI and motion artifacts in experimental videos.
- Reconstruction Time: While oversampling significantly improves depth quality, it increases reconstruction time (e.g., approximately 200s for a 32-bit code without oversampling, and 1800s with 10x sampling).
Limitations and Future Work
- Reconstruction Time: The primary limitation is the increased computational time required for image reconstruction, especially with higher oversampling ratios. The authors note that parallel processing was not utilized, suggesting this as a potential area for acceleration.
- Ambient Light Sensitivity: While not explicitly detailed as a limitation in the conclusion, the nature of ToF sensors generally implies some vulnerability to strong ambient light, which could affect signal-to-noise ratio.
- Crosstalk: Some crosstalk was observed between objects, potentially due to differences in measured vs. actual IRFs.
- Future Work: The authors are exploring longer exposure codes for higher compression ratios and developing image reproduction algorithms based on deep neural networks to reduce processing time. Further research into improving compression ratios and reducing image reproduction time is ongoing.
Summary updated