Time-of-flight cameras measure distance by timing how long their own infrared light takes to come back. When light bounces off a wall, then a floor, and then returns, several echoes add together in the same pixel. The camera then reports a wrong distance. This effect is called multi-path interference, and it rounds off corners, makes shiny floors look sunken, and is a known cause of robots misjudging obstacles. Neural networks now correct much of this error, but nobody can check a particular correction after the fact. If a robot hits a corner, a log can show which model ran on which frame. It cannot show whether the model's corrected depth was physically consistent with what the sensor measured.
We propose that the depth-correction network output a small, checkable explanation along with its depth map. For each image tile, it claims one direct echo and at most one indirect echo, each with a distance and a strength. A few times per second the camera captures an "audit frame" at three modulation frequencies instead of one. Anyone holding the logged raw measurements can recompute how well the claimed echoes reproduce them, using a closed-form physics formula and no neural network. Commitments to these raw frames, claims, and checks go into a tamper-evident append-only log. The same physics check also serves as an unlabeled training signal. The rate of audit frames is the main energy setting, and every unit of illumination energy is recorded. The point of the design is that the check is cheap to verify, so it does not require solving the depth problem again.
Core Claim
AUDITA builds signed, Merkle-committed logs (records chained by hashes, so any later edit changes a single root hash) and assigns causal blame among agents. Its Theorem 4 shows that an auditor limited to logged evidence cannot judge whether an outcome was correct unless it can solve the underlying task. We accept that result and change what gets logged. Time-of-flight depth has an unusual property: the sensor's forward physics is known in closed form. Echoes add as sinusoids, as described by Sarbolandi et al.. Checking a proposed set of echoes against raw measurements is therefore a cheap evaluation, while finding those echoes is a hard inverse problem.
The claim is that a three-frequency audit frame turns each tile's correction into a verifiable certificate. We test whether that certificate predicts real depth error and catches tampered or degraded models better than the model's own confidence does, at a fraction of the illumination energy of capturing every frame at multiple frequencies.
Let $m_f \in \mathbb{C}$ be the raw complex measurement (amplitude and phase) of a tile at modulation frequency $f$. Let $(a_k, d_k)$ be the claimed strength and distance of echo $k \le 2$. Let $g_f(d) = e^{-i 4\pi f d / c}$ be the ideal single-echo response, where $c$ is the speed of light, and let $\hat{g}_f$ be its per-sensor calibrated version. The certificate residual is
$$r = \sum_{f \in \{f_1,f_2,f_3\}} \Big| m_f - \sum_{k=1}^{2} a_k\, \hat{g}_f(d_k) \Big|^2 .$$
Three frequencies give six real equations against four unknowns. That surplus is why the residual means anything. With only two frequencies, a two-echo explanation can almost always be fit exactly, so a near-zero residual would carry no information.
Research Program
The method has three parts.
The first part is the model. We start from a single-frequency correction network in the style of MTDNet, which uses camera motion across two frames instead of multi-frequency sweeps and predicts confidence maps. We add a second output that predicts tile-level echo claims $(a_k, d_k)$. On audit frames, the claims are trained to drive $r$ toward zero, and the direct-echo distance $d_1$ is tied to the dense depth output.
This training signal is what separates us from the correlation-consistency loss of Wang, Wang & Wen. Their loss asks a single predicted depth to reproduce the raw correlations, even though MPI corrupts those correlations. That is the setting where HAMMER found models learning to reproduce sensor bias. HAMMER reports an overall RMSE of 113.29 mm for raw-iToF supervision against 20.70 mm for ground-truth supervision. Our loss instead explains the raw signal as direct plus indirect echoes, so the network is never pushed to put the mixed phase into the depth map. Which loss avoids the trap is an empirical question, and we test it directly.
Labels enter in one narrow place. A small labeled set calibrates $\hat{g}_f$ against non-ideal waveforms and sets acceptance thresholds for $r$. We measure how few labeled frames this needs. This targets the mismatch between ideal and real sensor physics that leaves residual sim-to-real error in DeepToF and in Agresti et al..
The second part is the log. Every frame commits a hash of its raw correlations and of the model weights. Audit frames additionally commit their tile claims and residual map. Records are batched into Merkle trees, and only the roots are anchored externally. This follows FG-Trac, which reports under 2% epoch overhead for CNNs but about 1.2 GB of off-chain storage per 100 CIFAR-10 epochs. We do not log per-sample gradient influence, since that causes FG-Trac's 2.5x–2.9x slowdowns. We log only physics claims at tile resolution.
The audit schedule is committed to the log as well. It has a policy-enforced minimum rate, plus extra audits triggered when the model's confidence falls. The confidence is untrusted, so it can only add audits, never remove them. The committed schedule doubles as a record of emitter-on time. That gives a concrete instance of the verifiable energy measurement that AI Cap-and-Trade identifies as missing.
The third part is accounting. We report joules per certified frame, counting both illumination time and compute, as a function of audit rate. This trades off energy against the longest stretch a failure can go uncertified.
Baselines
The naive reference is the raw single-frequency sensor depth with ordinary logging. It sets the floor for accuracy. Under tampering, it is also the floor for auditability, since AUDITA reports that unsigned logs let forgers frame innocent parties in 100% of probes.
The strongest accuracy competitor is MTDNet itself, trained on the same unlabeled data and the same label budget, with an equal hyperparameter search. It reports MAE 0.82 cm on TFT3D and 0.49 cm on the Cornell Box, according to MTDNet. Its logged confidence map is the competing audit signal. We count our method as beating it if certificate residuals rank erroneous pixels clearly better than MTDNet confidence does, measured by area under the ROC curve for flagging pixels above an error threshold, while staying within 10% of its MAE.
For domain adaptation, we compare against Agresti et al. on their S4 and S5 data. They report S4 MAE of 2.36 cm, against 3.19 cm for their synthetic-supervised model and 5.43 cm raw. The goal there is to match 2.36 cm while using multi-frequency light only on audit frames.
The physics-only competitor is always-on multi-frequency capture with sparse unmixing, following Freedman et al.. It is run on identical scenes and compared on accuracy and joules per frame. No reported energy result was found on Lacuna for this method or for any of the learned baselines, so we will measure energy for all of them ourselves.
Two ablations isolate the key ingredients. The first replaces our decomposition loss with Wang et al.'s single-path correlation loss. The second drops the third frequency, which is expected to make the certificate uninformative.
Why not just…?
Why not just hash the raw frames and model, and let auditors replay the network? Replay proves which output was produced, not whether that output was physically consistent. This is exactly the completeness barrier in AUDITA. Our tampering experiment tests the difference. We swap in a model fine-tuned to smooth corners while keeping its confidence high. Replay will faithfully reproduce the bad depth, and only a physics check can reject it.
Why not just log the network's confidence map? Confidence is the model reporting on itself. A degraded or adversarial model controls it, and nobody can recompute it independently. This is a real threat on accuracy, because MTDNet's confidence may correlate well with error on benign data. The head-to-head AUROC comparison settles that for benign data. The tampered-model condition settles it for the adversarial case.
Why not just run multi-frequency unmixing on every frame? That approach needs no learning and gives a certificate everywhere. Its cost is about triple the illumination time, as noted for frequency-diversity methods in Resolving Multi-path Interference..., and it struggles on non-Lambertian surfaces. If its accuracy and energy come out close to ours at reasonable audit rates, the learned component is not worth it. We would then report that outcome.
Why not just train on more path-traced synthetic data? Agresti et al.'s 3.19 cm versus 2.36 cm comparison shows that synthetic supervision alone leaves a gap on real sensors. More importantly, synthetic data provides no certificate at deployment time.
Proposed Experiments
We evaluate accuracy on TFT3D, the Cornell Box, Agresti's S4 and S5 sets, and HAMMER. HAMMER's scanned ground truth separates reflective objects, where dToF supervision gives 71.59 mm against 36.75 mm for ground-truth supervision. Where public sets lack three-frequency raw data, we capture our own audit sequences on a commodity sensor that can switch frequencies, with a small HAMMER-style labeled subset for calibration.
Four checks decide the outcome. First, residual-versus-error AUROC must exceed the confidence baseline overall and especially on reflective tiles. Second, the false-acceptance rate must be measured: how often a wrong two-echo claim passes the threshold. This is the main risk to the idea, since nonlinear echo fitting may admit wrong solutions that still fit. Third, we plot label-efficiency curves for calibration with 0 to 200 labeled frames. Fourth, we plot energy against uncertified time window across audit rates, alongside storage per hour. Success means the certificate catches tampered models that replay misses, at substantially less illumination energy than always-on multi-frequency capture, with accuracy near MTDNet.
References
- AUDITA: certified auditing and causal attribution of adverse outcomes in autonomous multi-agent systems
- Kinect range sensing: Structured-light versus Time-of-Flight Kinect
- Exploiting Dual-Correlation for Multi-frame Time-of-Flight Denoising
- Self-supervised End-to-end ToF Imaging Based on RGB-D Cross-modal Dependency
- Is my Depth Ground-Truth Good Enough? HAMMER - Highly Accurate Multi-Modal Dataset for DEnse 3D Scene Regression
- DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging
- Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning
- Fine-Grained Traceability for Transparent ML Pipelines
- AI Cap-and-Trade: Efficiency Incentives for Accessibility and Sustainability
- SRA: Fast Removal of General Multipath for ToF Sensors
- Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization