Cheap depth cameras that measure distance from the phase of reflected modulated light, called indirect time-of-flight (iToF) cameras, cannot tell an object at 2 m from one at 7 m when the modulation wavelength is 5 m. They see only the phase, and the phase repeats. The standard pipeline first guesses the missing integer "cycle count" for every pixel. This step is called phase unwrapping. The pipeline then turns the depth map into a 3D point cloud and hands it to a recognition network. When the guess is wrong, the error is not small jitter. A whole surface patch jumps along the viewing ray by an exact multiple of the wavelength and appears as a "ghost" copy of geometry. Point cloud classifiers are trained on clean synthetic CAD models and hardened only against local noise and missing points, so they have never seen such ghosts.
We propose to keep the ambiguity in the data instead of resolving it early. For each pixel, the camera's phases at several frequencies admit a small set of candidate depths, each with a likelihood that a physical sensor-noise model can compute. We feed all candidates, as a weighted multi-hypothesis point set, to a 3D recognition network trained on synthetic objects rendered through a physically simulated iToF measurement process. The network learns which candidate surfaces form a plausible object. The shape prior of the recognition task then does the unwrapping, in exactly the regime where local spatial-continuity unwrappers have no neighbors to lean on: long range, low signal, few frequencies, and isolated objects whose boundaries are depth discontinuities.
Core Claim
Committing to a single unwrapped depth before recognition throws away information the recognizer could have used. The sim-to-real gap for iToF recognition is better closed at the level of the raw wrapped measurement than at the level of the point cloud. Concretely, write $\theta_j$ for the wrapped phase at modulation frequency $f_j$, $c$ for the speed of light, and $\mathbf{r}_i$ for the unit viewing ray of pixel $i$. Candidate points are
$$\mathbf{p}_{ik} = \frac{c}{4\pi f_1}\,(\theta_1 + 2\pi k)\,\mathbf{r}_i, \qquad w_{ik} \propto \prod_j \mathcal{N}\!\left(\mathrm{wrap}(\theta_j - 4\pi f_j d_{ik}/c);\,0,\,\sigma_{ij}^2\right),$$
Here $k$ indexes integer cycle counts up to the sensor's maximum range, $d_{ik}$ is the candidate depth, and $\sigma_{ij}$ is the per-pixel phase noise. That noise is predicted from amplitude and ambient light with a heteroscedastic shot-plus-read model, so $w_{ik}$ measures how consistent candidate $k$ is with all frequencies. The network consumes the set of points $\mathbf{p}_{ik}$ together with the weights $w_{ik}$. A soft selection head predicts a distribution over $k$ per pixel, and a standard backbone classifies the object. The prediction is that this beats every unwrap-then-classify pipeline, and that the margin grows as signal-to-noise ratio and the number of frequencies shrink.
Where This Sits
The candidate-generation step comes from Efficient Multi-frequency Phase Unwrapping Using Kernel Density Estimation. That paper enumerates up to 30 depth hypotheses from the Kinect v2's 16/80/120 MHz phases, weights them with a "circle-tangent" noise model, and picks one per pixel by spatial kernel density over a local window. We reuse the hypotheses and the noise weighting. We replace the hand-crafted local selection with selection driven by object shape, and we do not commit to one candidate before recognition.
Three learned depth systems motivate the gap by stopping at 2.5D pixel error. Deep End-to-End Time-of-Flight Imaging (ToFNet) maps raw two-frequency correlations to depth with a U-Net trained on 100k pbrt-v3 transient renders, without modeling heteroscedastic noise. Wild ToFu resolves cycles using RGB context at a coarse bottleneck. Learnable Burst-Encodable Time-of-Flight Imaging removes wrapping in hardware with an expensive image intensifier. All three output one depth per pixel and never test downstream 3D recognition. We test recognition directly, and we need no RGB camera or special hardware.
On the recognition side, Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions (AdaptPoint) learns shape-conditioned local deformations and masks. Quasi-Balanced Self-Training on Noise-Aware Synthesis shows that Gaussian jitter fails because real noise comes from structured matching failures. Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation gets realistic artifacts by simulating the actual decoding algorithm. These papers supply the methodological lesson: simulate the sensor, not the symptom. None of them model modulo-phase physics, and AdaptPoint's local operations cannot represent a patch translated by a full wavelength.
Robust Phase Unwrapping via Deep Image Prior constrains our design. It reports that a supervised unwrapper trained on synthetic shapes drops to 22.01 dB on real-like speckled organoids, against 43.52 dB for test-time optimization. That is a warning that our synthetic-trained selector could overfit too. We address it with the noise-model ablation and the real-capture test below, rather than assuming transfer.
Research Program
The first component is a wrapped-measurement simulator for object datasets. We render ModelNet40 and ShapeNet objects, and also real geometry from ScanObjectNN, ScanNet and NYU-Depth V2, as raw multi-frequency correlation images. The renderer includes $1/d^2$ falloff, surface albedo, ambient infrared, Poisson shot noise plus Gaussian read noise (the heteroscedastic model used in Optical Flow in the Dark), and frequency sets matching the Kinect v2 and a two-frequency Lucid Helios-like configuration. Each object is placed at a sweep of ranges, so the same shape appears from easy (one cycle, high SNR) to hard (many cycles, low SNR). We keep multipath interference out of the main study and treat it only as a stress test. That keeps the claim about wrapping clean.
The second component is the multi-hypothesis recognizer. The candidate points with weights $w_{ik}$ as an extra feature go into PointNet++, PointBERT and PointMLP backbones. A lightweight per-point head outputs a distribution over $k$, and its expected point feeds a second, standard recognition pass. Training uses the classification loss, plus cross-entropy on $k$ where synthetic ground truth exists. The point budget is held equal to the baselines by subsampling candidates in proportion to $w_{ik}$.
The third component is analysis. We measure how recognition accuracy depends on SNR, the number of frequencies, and object size relative to the wavelength, and we record per-pixel cycle accuracy as a byproduct. This also yields a by-product the community lacks: corrupted versions of ModelNet40 and ScanObjectNN that contain wrap errors, in the style of ModelNet-C, but produced by measurement physics.
Baselines
The naive reference point is vendor-style per-pixel Chinese Remainder Theorem unwrapping: integer reconstruction from several frequencies, applied independently per pixel. It is followed by statistical outlier removal and a PointNet++ classifier trained with standard augmentation on clean CAD. It represents what a practitioner gets today. Lawin et al. show this unwrapping produces salt-and-pepper cycle errors under noise, so we expect it to set the floor.
The strongest unwrap-then-classify competitor is KDE unwrapping (Lawin et al.). It uses the same candidates and the same noise model as our method, which makes it the fairest test of whether committing early costs anything. Its reported real-world result is 52% more valid measurements than libfreenect2 at a 1% outlier rate on the lecture-hall scene. A learned variant is ToFNet retrained on our simulator, as a supervised depth front end. Its downstream classifier is the same backbone trained on its outputs over simulated training data, with an identical tuning budget.
The strongest robustness-by-augmentation competitor is AdaptPoint on top of KDE-unwrapped clouds. It reports an mCE of 63.7% with PointNet++ on ModelNet-C (down from 107.2%) and 78.3% on ScanObjectNN-C. We also include ULIP-pretrained PointMLP, reported at 89.4% on clean ScanObjectNN, and SimpleView, reported at 79.5% on ScanObjectNN. Together they test whether 2D anchoring or projection already absorbs ghosts.
No reported result found on Lacuna for any method on wrap-corrupted point clouds, so all baseline numbers on the new benchmark will be produced here. Every method sees identical simulated training objects, identical point budgets, and the same hyperparameter search budget. Beating the field means higher accuracy than the best unwrap-then-classify pipeline at matched compute in the low-SNR, few-frequency bins, with no loss in the easy bins.
Three ablations isolate the key ingredient. The first gives our network only its own top-weighted candidate, the committed version with the same simulator and backbone; this isolates the value of keeping alternatives. The second replaces $w_{ik}$ with uniform weights; this isolates the noise model. The third trains with Gaussian phase noise instead of the heteroscedastic simulator; this isolates measurement-level realism.
Why not just…?
Why not just add ghost-surface augmentation to ordinary point clouds? This is the cheapest rival and a real one. We implement it directly: random surface patches are duplicated or displaced by multiples of $c/2f$, with displacement probability tuned to match the simulator's error rates. It teaches invariance to ghosts. It cannot recover the true surface when the committed unwrapper chose the wrong cycle for most of a small object, because the correct geometry is no longer in the input. The decisive comparison is in the low-SNR bins where the committed cycle is wrong for a majority of object pixels. If ghost augmentation matches us there, the hypothesis-set idea is unnecessary and we will say so.
Why not just unwrap better first, with KDE or RGB fusion, and then classify? Better unwrappers use local spatial continuity, which fails at object silhouettes. There, depth is discontinuous and a ghost patch is locally self-consistent. Wild ToFu needs a calibrated RGB camera and still reports 0.56 m RMSE in real scenes, an error large compared with a tabletop object. The KDE and retrained ToFNet baselines above settle this. If either closes the gap across the sweep, the conclusion is that recognition-driven unwrapping adds nothing, and that would be a useful negative result.
Why not just rely on 2D anchoring like ULIP or SimpleView, which are already robust to real noise? Their robustness was measured against jitter, occlusion and clutter, not against displaced copies of surfaces. A ghost along the viewing direction overlaps the true silhouette in projection. Running both on the wrap benchmark answers whether that matters in practice.
Why not just use direct time-of-flight hardware? As Bao et al. note, it needs picosecond timing electronics or intensifiers. Commodity CMOS iToF remains the cheap option, which is the setting this work targets.
Proposed Experiments
The synthetic study trains on simulated ModelNet40 and tests on simulated ModelNet40 and ScanObjectNN geometry across range and SNR bins. It reports overall accuracy and per-pixel cycle accuracy. Real-geometry transfer renders ScanNet and NYU-Depth V2 surfaces through the simulator and evaluates detection-style object accuracy on wrapped crops. The real-measurement test has two parts. First, the per-pixel selection head runs on raw Wild ToFu correlations, compared on RMSE against the reported 0.56 m (Wild ToFu) and 0.99 m (ToFNet), without using RGB. Second, we capture a modest new set of everyday objects with a Kinect v2 through libfreenect2 raw access, at ranges from 4 to 15 m, with ground truth from close-range scans and object labels. We report overall accuracy against all baselines, unchanged from simulation.
The decisive check is the gap between our method and its committed-hypothesis ablation on real captures in the far-range bins. If the gap is positive and grows with range, while ghost augmentation and KDE-then-classify stay below it, the claim holds. If the gap vanishes on real data while it remains in simulation, the selector has overfit synthetic physics, the PUDIP warning has come true, and the next step would be test-time adaptation of the selection head.
References
- Efficient Multi-frequency Phase Unwrapping Using Kernel Density Estimation
- Deep End-to-End Time-of-Flight Imaging
- Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments
- Learnable Burst-Encodable Time-of-Flight Imaging for High-Fidelity Long-Distance Depth Sensing
- Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions
- Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap
- Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation
- Robust Phase Unwrapping via Deep Image Prior for Quantitative Phase Imaging
- Optical Flow in the Dark