Noise-Calibrated Change Gating for Wearable Depth Streams

A blind pedestrian's wearable depth camera sees almost the same sidewalk thirty times a second. In principle that redundancy is free speed. A network could recompute only the parts of the scene that changed and reuse everything else, which is how "event-driven" video transformers cut compute by 2–4×. The difficulty is that cheap depth sensors are noisy over time even when nothing moves. Stereo matching flickers, speckle patterns decode differently frame to frame, and returns drop out at grazing angles. A rule that says "recompute what changed" therefore wastes its budget on sensor noise. Raising the threshold to ignore that noise also hides the small, quiet signals that matter most to a blind walker, such as the lip of an open drain or a down-step.

This note proposes a different change test. A patch is declared changed only when its frame-to-frame difference exceeds what the specific sensor would produce on that geometry, at that range and angle, under the wearer's current motion. The expected noise comes from a physics-based simulator of the sensor, calibrated against a handful of real captures and extended to render temporally coherent noise along walking trajectories. The simulator provides the one training signal real data cannot: many independent noise realizations of the same moving scene. We use them to teach a small network to predict, per image patch, how much feature change is "just the sensor." The application is negative-obstacle detection on wearable stereo depth, where both latency and missed drop-offs are dangerous.

Core Claim

Change-gated computation on wearable depth streams fails for a specific, fixable reason: the gate uses an unnormalized distance. Eventful Transformers keeps a buffer of reference tokens, the small feature vectors a vision transformer computes for each image patch. It recomputes only the top-$r$ tokens whose $L_2$ distance from their reference is largest. It reports a 3.8× GFLOP reduction on ImageNet VID at a 1.68% mAP drop, but it was tested only on clean RGB video. We claim that replacing this distance with a noise-normalized, ego-motion-compensated score will recover most of that compute saving on depth streams without suppressing negative obstacles. The score for patch $i$ at time $t$ is

$$ s_i^t = \frac{\lVert x_i^t - \mathcal{W}_t(r_i) \rVert^2}{\hat{\sigma}_i^2(d, \theta, \omega) + \epsilon} $$

The symbols are defined as follows. $x_i^t$ is the current token. $r_i$ is its buffered reference. $\mathcal{W}_t$ warps references by the head motion measured by the inertial sensor. $\hat{\sigma}_i^2$ is the predicted feature-space noise variance given the patch's depth $d$, surface incidence angle $\theta$ and angular velocity $\omega$. $\epsilon$ is a small constant. The comparison point is unchanged gating, where the denominator is 1 and there is no warp.

The new part is not change gating itself, which Eventful Transformers and DyCoke already do with $L_2$ and cosine thresholds. The new part is a denominator learned from physics-simulated noise realizations, which neither of those methods nor any physics simulator in the file attempts.

Research Program

The work has three pieces.

Temporal sensor simulation. Physics-based depth simulators already reproduce the structured noise of active stereo. Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation ray-traces projected patterns and runs the real decoder. Mesh2Point / QS³ projects speckle and runs stereo matching. DDS makes the whole pipeline differentiable, so sensor parameters can be fit by gradient descent to real scans. All three simulate isolated static frames. These papers supply our method: we reuse DDS's differentiable block matching to fit a stereo sensor's parameters to roughly a hundred static real captures. We then extend it in time. We render sidewalk scene meshes along chest-camera trajectories driven by recorded gait inertial traces, with motion blur and rolling-shutter timing. For each trajectory we render $K$ independent noise realizations of the same clean geometry, varying projector speckle phase, sensor read noise and matching ties. This gives, for each patch and time, an empirical spread of tokens produced by noise alone.

A noise head. A small convolutional head reads the raw depth patch, its valid-pixel mask, the estimated incidence angle and the inertial angular rate. It is trained to regress the per-token variance $\hat{\sigma}_i^2$ observed across the $K$ realizations, passed through the frozen patch embedding and early transformer layers. The same head also outputs a per-pixel probability that the sensor returns no depth. This second output addresses the ambiguity raised by SurDis and the pRGB-D water-hazard work: a hole in the depth map can mean a drop-off or a sensor failure. If depth is missing where the model expects valid returns, and stays missing across warped frames, that is evidence of a real discontinuity. If depth is missing where dropouts are expected, as at specular or grazing surfaces, it is weak evidence.

Gated detector. A ViTDet-style detector, the backbone Eventful Transformers evaluated, is wrapped with redundancy-aware attention using the score above. It is fine-tuned on a mix of simulated sequences and real SurDis frames. The token budget $r$ is either fixed or set by thresholding $s_i^t$. The second option has a statistical reading: under Gaussian-like noise, $s$ behaves like a chi-squared statistic, so one threshold gives a roughly fixed false-change rate across ranges and surfaces.

The application papers motivate the latency target. HIDA needs 8.8 s per scan-then-plan cycle. FDPN runs at 1.7 FPS, which its authors call too slow for guidance. ERF-PSPNet is fast at 82 FPS on a GTX1050Ti but processes every frame independently.

Baselines

The naive reference is the same detector run densely on every frame with no gating. It sets the accuracy ceiling and the compute cost to beat. Alongside it we report the SurDis paper's frame-wise YOLOv4, which reaches mAP 0.427 on SurDis (SurDis), as the published reference on this data. The SurDis mAP threshold and split are taken from that paper.

The strongest existing method is Eventful Transformers' $L_2$ top-$r$ gate applied to the same backbone and data (Eventful Transformers). It is the most direct competitor because it already does token-level reuse with delta attention. The only question is whether its distance measure survives depth noise. For fairness, the gate and our gate share weights, training schedule and the same grid of $r$ values. Each gets the same hyperparameter search budget over thresholds, and we compare full accuracy–compute curves rather than single points.

Two ablations isolate the key ingredient. The first is ego-motion warp without the noise denominator, to separate what motion compensation buys from what noise calibration buys. The second replaces the physics-trained head with an empirical lookup table: per-pixel temporal variance measured on static real captures and binned by depth only. This tests whether the simulator's geometry-dependent noise matters at all.

A third ablation trains the noise head on Gaussian-plus-random-dropout renders instead of physics renders. The bin-picking work found domain randomization reaches real IoU 0.584 against 0.648 for physics simulation (Close the Sim2real Gap). We expect a similar ordering here, but it must be shown on streaming depth.

Beating the strongest baseline means one of two things. Either we match dense-detector mAP within the same tolerance Eventful Transformers accepts on ImageNet VID, about 1.7 points, at a clearly lower recomputed-token fraction. Or, at equal compute, we show higher recall on SurDis's drop-off and drainage classes together with earlier first detection.

Why not just…?

Why not just smooth the depth stream before gating (temporal median or exponential averaging)? This is the most serious threat, because it is nearly free and kills flicker. It fails in two ways we can measure. Smoothing delays genuine changes by the filter length, which costs lead distance on obstacles approached at walking speed. A median also fills persistent holes with neighbouring values, erasing the "missing depth that should not be missing" signal that marks an open drain. We include a median-prefiltered Eventful baseline with filter length tuned on validation data. If it matches our method on first-detection distance and negative-class recall, the noise head is unnecessary and we will report that.

Why not just learn the gate end-to-end on real sequences with a compute penalty? A learned gate can absorb whatever noise the training sensor has, and it is a real alternative. Its weakness is that real streams never contain repeated noise realizations of the same moving scene. The gate therefore learns noise and motion entangled with the training sensor, and nothing tells it what the noise level is under conditions it has not seen. We test this by training on one sensor and evaluating on another using the cross-system protocol of RGB-D-SS. The physics route only needs a re-fit of DDS parameters to static captures from the new sensor. If the learned gate transfers equally well, the simulator earns no place in the method.

Why not just run a fast per-frame CNN, or subsample frames? ERF-PSPNet already reaches 82 FPS on a laptop-class GPU, which undercuts a pure speed argument on that hardware. Our case rests on two other points. The first is compute and energy on phone- or glasses-class processors, where we measure wall-clock time directly. The second is temporal stability of warnings, where per-frame networks flicker. Frame subsampling to 2–5 FPS leaves 200–500 ms blind intervals, during which a pedestrian at 1.2 m/s moves a quarter to over half a meter. We include a subsampled-dense baseline at matched compute and score it on first-detection distance.

Why not just fit planes with RANSAC? Geometric plane fitting showed much higher cross-system variation than learned segmentation in RGB-D-SS. It also cannot see water, as the 60.6% expansion error before polarization cues in pRGB-D shows. We keep a RANSAC drop-off detector as a cheap reference but do not expect it to compete on irregular SurDis terrain.

Proposed Experiments

The primary benchmark is SurDis, whose 200 stereo sequences give both temporal context and labels for the five discontinuity classes. Where the raw stereo pairs are available, we re-run block matching ourselves so that the simulator and the real pipeline use the same matcher. We also collect a small calibration-and-test set with the same class of stereo sensor: about a hundred static captures for DDS fitting, plus walking sequences with inertial logs. Cross-sensor transfer uses the four wearable systems of RGB-D-SS. Eventful Transformers' own ImageNet VID number serves only as a reference that the reimplementation is correct, not as a target on depth.

Metrics are SurDis mAP overall and on negative classes, fraction of tokens recomputed and GFLOPs, and wall-clock latency on an edge board. Two further metrics are first-detection distance per obstacle instance, recovered from the frame index of the first correct detection and walking speed, and alert flicker, the number of on–off toggles per true obstacle. All sequences are scored per instance with seeds varied.

The decisive check is a static-scene probe. The wearer stands still, or a recorded sequence is replayed with the scene held fixed. Here every recomputation is spurious by construction. Unnormalized gating should spend a large share of its budget on flicker, while our gate should spend little and keep a roughly constant false-change rate across range bins. If the noise-normalized gate does not lower spurious recomputation here, the core claim is wrong regardless of downstream mAP.

The second decisive check is the dropout-versus-drop-off test. On SurDis drainage and drop-off instances, plus a held-out set of specular and glass surfaces, we measure whether the predicted dropout probability separates true discontinuities from sensor failures better than raw hole size does. The last check is the empirical-lookup-table ablation. If depth-binned real variance does as well as the physics-trained head, the honest conclusion is that noise normalization matters but simulation does not, and the paper would say so.