D2ToF: One-Step Diffusion for Depth Super-Resolution from Lightweight Single-Photon LiDAR Waveforms

Pallabjyoti Deka, Lewis Wilson, Stephen McLaughlin, Jonathan Leach, Abderrahim HalimiarXiv 2026 · arXiv

TL;DR

D2ToF is a novel one-step diffusion framework that super-resolves ultra-low-resolution (8×8) LiDAR waveforms into high-fidelity 128×128 depth maps in real-time. By distilling a multi-step diffusion teacher and incorporating a physical data-consistency term, it achieves state-of-the-art accuracy at speeds 130× faster than traditional diffusion models.

What Problem Does This Paper Address?

Single-photon LiDAR (SPL) is a powerful tool for depth sensing, offering high sensitivity and timing precision. However, high-resolution SPL arrays are prohibitively expensive. Most consumer-grade or cost-effective sensors (like those found in smartphones) provide extremely sparse data—often just an 8×8 grid of pixel waveforms.

Upsampling this data to a usable resolution (e.g., 128×128) is a massive challenge. It’s an "extreme" super-resolution problem where a single 8×8 pixel might cover a large, complex area of a scene. Existing methods either produce over-smoothed results, fail in low-light conditions, or use iterative generative models (like standard diffusion) that are far too slow for real-time applications like autonomous navigation or AR/VR.

Methods

The authors propose D2ToF (Distilled Diffusion for Time-of-Flight), which uses a two-stage approach to bridge the gap between high-quality generative modeling and real-time execution:

  1. Stage 1: Conditional Diffusion Teacher: They first train a multi-step diffusion model to learn the distribution of high-resolution depth maps. This model is conditioned on a "triad" of inputs: the raw low-resolution ToF histograms, a high-resolution reflectivity image, and a relative depth estimate from a frozen foundational model (DepthAnything V2).
  2. Stage 2: One-Step Distillation: To achieve real-time speed, they distil this multi-step teacher into a one-step student generator. They use an amortized variational distillation strategy, which allows them to bake a physical "data-consistency" term directly into the training objective.
Overview of the two-stage training and distillation process.

A key innovation is the differentiable histogram rendering layer. During training, the model renders its predicted high-resolution depth back into a low-resolution histogram and penalizes any deviation from the actual sensor measurements. This "anchors" the generative model to the physical reality of the captured data.

Fig. 4. Overview of the distillation stage. A one-step generator qϕ(x0 | y) is trained so that its samples (i) match the learned prior via teacher-student score matching and (ii) enforce estimated histogram Hest to be consistent with the measured histogram H.

Datasets

  • Training: The models were trained on synthetic depth-reflectivity pairs from the Hypersim (43,140 frames) and MPI Sintel (2,128 frames) datasets.
  • Evaluation (Simulated): 12 clean scenes from the Middlebury dataset were used to benchmark against state-of-the-art baselines.
  • Evaluation (Real): The authors validated the model using real-world data captured with a low-cost STMicroelectronics VL53L8 SPAD sensor (8×8 mode), using a Kinect sensor to provide guidance images and ground truth for comparison.

Key Insights and Ideas

  • One-Step is Sufficient: Through distillation, the model can jump from noise to a clean depth map in a single forward pass (~83ms) without sacrificing the structural detail typically associated with slow, multi-step diffusion.
  • Physics-Grounded Generative Modeling: By incorporating the sensor's forward model (histogram rendering) into the loss function, the model becomes significantly more robust. Even if the guidance image is noisy or degraded (e.g., in low light), the histogram data keeps the reconstruction accurate.
  • The Power of Multimodal Priors: Monocular depth models (like DepthAnything) are excellent at understanding scene geometry but lack absolute scale. LiDAR has perfect scale but poor resolution. D2ToF uses the monocular estimate as a "geometric glue" to help the network intelligently upsample the sparse LiDAR data.

Key Results

  • Speed: D2ToF performs inference in approximately 83 milliseconds on an NVIDIA RTX 3090, making it roughly 130× faster than its own multi-step teacher and significantly faster than other diffusion-based competitors like Marigold-DC.
  • Accuracy: On the Middlebury benchmark, D2ToF achieved an average RMSE of 11.7 mm, a massive improvement over baselines like DELTAR (43.6 mm) and DVSR (40.7 mm).
Fig. 1. Reconstruction quality (RMSE in mm) vs. time (sec.) of our proposed method in comparison to DELTAR [8], DepthAnything-V2 [9], DVSR [10], and Marigold-DC [11] evaluated on 12 clean scenes of Middlebury dataset.
  • Qualitative Superiority: As shown in Figure 5, D2ToF successfully recovers sharp object boundaries and fine details (like the intricate shapes in the "Art" scene) that other methods either blur or miss entirely.
Qualitative comparison showing D2ToF's ability to recover sharp details compared to other methods.

Limitations and Future Work

  • Hardware Requirements: While 83ms is fast, it was measured on a high-end desktop GPU. Deploying this on mobile hardware (where these low-cost sensors are actually used) would likely require further model compression or optimization.
  • Co-registration: The method assumes a well-aligned high-resolution reflectivity image is available. While the authors added random misalignment during training to improve robustness, extreme offsets between the camera and LiDAR could still pose challenges.
  • Future Directions: The authors suggest exploring multi-frame consistency for video sequences and adapting the framework to handle more complex "multi-path" interference where light bounces off multiple surfaces before returning to the sensor.

Summary updated

Abstract

Single-photon LiDAR (SPL) sensors offer exceptional temporal resolution and single-photon sensitivity for depth imaging. However, cost-effective SPL arrays typically suffer from extremely low spatial resolutions (e.g. $8\times 8$ pixels), limiting their real-world utility. In this paper, we propose a novel, one-step super-resolution framework, D2ToF, Distilled Diffusion for Time-of-Flight, that bridges the gap between high-fidelity generative modeling and real-time execution. Our approach uses an amortized variational distillation strategy that allows for a measurement-based data consistency term to be incorporated directly into the one-step training objective. This encourages that the synthesized high-resolution depth maps are consistent with the captured low-resolution Time-of-Flight (ToF) histograms. To overcome the severe ill-posedness of upsampling extremely degraded inputs, we introduce a multimodal guidance strategy. By leveraging a foundational monocular depth model to extract structural priors from a high-resolution reflectivity image, our network is guided by a rich triad of the ToF histogram, reflectivity map, and relative depth estimate. Experimental results outperform evaluated baselines, achieving $16\times$ spatial super-resolution (from $8\times 8 \times 100$ histograms to $128\times 128$ depth maps) at an accelerated inference speed of approximately $83$ milliseconds.
BibTeX
@misc{deka2026d,
  author = {Pallabjyoti Deka and Lewis Wilson and Stephen McLaughlin and Jonathan Leach and Abderrahim Halimi},
  title = {{D2ToF: One-Step Diffusion for Depth Super-Resolution from Lightweight Single-Photon LiDAR Waveforms}},
  year = {2026},
  eprint = {2611.18243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18243},
}
Figure