TL;DR
D2ToF is a novel one-step diffusion framework that super-resolves ultra-low-resolution (8×8) LiDAR waveforms into high-fidelity 128×128 depth maps in real-time. By distilling a multi-step diffusion teacher and incorporating a physical data-consistency term, it achieves state-of-the-art accuracy at speeds 130× faster than traditional diffusion models.
What Problem Does This Paper Address?
Single-photon LiDAR (SPL) is a powerful tool for depth sensing, offering high sensitivity and timing precision. However, high-resolution SPL arrays are prohibitively expensive. Most consumer-grade or cost-effective sensors (like those found in smartphones) provide extremely sparse data—often just an 8×8 grid of pixel waveforms.
Upsampling this data to a usable resolution (e.g., 128×128) is a massive challenge. It’s an "extreme" super-resolution problem where a single 8×8 pixel might cover a large, complex area of a scene. Existing methods either produce over-smoothed results, fail in low-light conditions, or use iterative generative models (like standard diffusion) that are far too slow for real-time applications like autonomous navigation or AR/VR.
Methods
The authors propose D2ToF (Distilled Diffusion for Time-of-Flight), which uses a two-stage approach to bridge the gap between high-quality generative modeling and real-time execution:
- Stage 1: Conditional Diffusion Teacher: They first train a multi-step diffusion model to learn the distribution of high-resolution depth maps. This model is conditioned on a "triad" of inputs: the raw low-resolution ToF histograms, a high-resolution reflectivity image, and a relative depth estimate from a frozen foundational model (DepthAnything V2).
- Stage 2: One-Step Distillation: To achieve real-time speed, they distil this multi-step teacher into a one-step student generator. They use an amortized variational distillation strategy, which allows them to bake a physical "data-consistency" term directly into the training objective.
A key innovation is the differentiable histogram rendering layer. During training, the model renders its predicted high-resolution depth back into a low-resolution histogram and penalizes any deviation from the actual sensor measurements. This "anchors" the generative model to the physical reality of the captured data.
Datasets
- Training: The models were trained on synthetic depth-reflectivity pairs from the Hypersim (43,140 frames) and MPI Sintel (2,128 frames) datasets.
- Evaluation (Simulated): 12 clean scenes from the Middlebury dataset were used to benchmark against state-of-the-art baselines.
- Evaluation (Real): The authors validated the model using real-world data captured with a low-cost STMicroelectronics VL53L8 SPAD sensor (8×8 mode), using a Kinect sensor to provide guidance images and ground truth for comparison.
Key Insights and Ideas
- One-Step is Sufficient: Through distillation, the model can jump from noise to a clean depth map in a single forward pass (~83ms) without sacrificing the structural detail typically associated with slow, multi-step diffusion.
- Physics-Grounded Generative Modeling: By incorporating the sensor's forward model (histogram rendering) into the loss function, the model becomes significantly more robust. Even if the guidance image is noisy or degraded (e.g., in low light), the histogram data keeps the reconstruction accurate.
- The Power of Multimodal Priors: Monocular depth models (like DepthAnything) are excellent at understanding scene geometry but lack absolute scale. LiDAR has perfect scale but poor resolution. D2ToF uses the monocular estimate as a "geometric glue" to help the network intelligently upsample the sparse LiDAR data.
Key Results
- Speed: D2ToF performs inference in approximately 83 milliseconds on an NVIDIA RTX 3090, making it roughly 130× faster than its own multi-step teacher and significantly faster than other diffusion-based competitors like Marigold-DC.
- Accuracy: On the Middlebury benchmark, D2ToF achieved an average RMSE of 11.7 mm, a massive improvement over baselines like DELTAR (43.6 mm) and DVSR (40.7 mm).
- Qualitative Superiority: As shown in Figure 5, D2ToF successfully recovers sharp object boundaries and fine details (like the intricate shapes in the "Art" scene) that other methods either blur or miss entirely.
Limitations and Future Work
- Hardware Requirements: While 83ms is fast, it was measured on a high-end desktop GPU. Deploying this on mobile hardware (where these low-cost sensors are actually used) would likely require further model compression or optimization.
- Co-registration: The method assumes a well-aligned high-resolution reflectivity image is available. While the authors added random misalignment during training to improve robustness, extreme offsets between the camera and LiDAR could still pose challenges.
- Future Directions: The authors suggest exploring multi-frame consistency for video sequences and adapting the framework to handle more complex "multi-path" interference where light bounces off multiple surfaces before returning to the sensor.
Summary updated