Range-Gated Skip Connections for Echo-Based Depth

When a robot emits a chirp and listens to the echoes, each moment of the recording corresponds to one distance: sound arriving $t$ seconds after emission bounced off something about $ct/2$ meters away, where $c \approx 340$ m/s. So the time axis of an echo spectrogram (a picture of sound energy over time and frequency) is, after deconvolution, an axis of range. The frequency axis carries different information: material, surface roughness, and the direction cues that the two ears pick up. Current networks that predict depth maps from echoes ignore this. They treat the spectrogram as an ordinary image, compress it into a single vector, and regress depth with a pixel-wise L1 loss. The result is depth maps that blur chair legs and table edges into the wall behind them.

We propose one mechanism that uses this structure directly. The depth decoder predicts, for every pixel, a probability distribution over discrete depth bins. Each bin maps to a known delay window in the echo. The skip connection that feeds audio evidence into that pixel reads only from the delay windows its own bin distribution points to, after frequency-only processing has turned each window into a feature vector. In plain terms, each pixel "listens" at the ranges it believes it occupies. A pixel on an object boundary can therefore hold two ranges at once instead of averaging them into a mid-air ghost value. This links four threads: the time–frequency asymmetry of spectrograms, axis-decoupled processing, edge-preserving discrete depth, and attention-gated skip connections. It does so through a single physical correspondence rather than by stacking four modules.

Core Claim

Tying the decoder's discrete depth bins to delay windows in the deconvolved echo gives a skip connection whose gate has physical meaning. We claim this reduces boundary smearing and "ghost pixels" in echo-based depth beyond what three simpler changes can achieve: swapping the loss to classification, swapping in an axis-decoupled audio encoder, or adding unconstrained cross-modal attention. A second, falsifiable claim concerns where the gain should appear. It should concentrate at depth discontinuities whose foreground and background lie in separable delay windows, and it should vanish when the bin-to-delay mapping is scrambled.

Why This Is Open

Echo-to-depth work so far takes three routes, and none uses range-indexed evidence at the pixel level. BatVision: Learning to See 3D Spatial Layout with Two Ears encodes binaural echoes to a latent vector and expands it with a U-Net generator under an L1 loss. Its follow-up, The Audio-Visual BatVision Dataset for Research on Sight and Sound, supplies the BV1/BV2 benchmark with an audio-only U-Net baseline. In both, the time axis is collapsed before any spatial decoding. Evaluating Visual-to-Echo Distillation for Binaural Depth Prediction beyond Simulations distills frozen visual teachers into an audio U-Net student. Its authors report an "over-smoothing bias" that flattens small obstacles into the background. That observation motivates our proposal, and we target that failure directly. AVS-Net uses cross-modal attention between a ResNet-18 audio encoder and a visual encoder to estimate a global metric scale. It explicitly does not correct local structure or boundary bleeding, and it names deeper integration of audio into the depth decoder as future work. Binaural SoundNet predicts coarse depth from passive binaural audio by regression with isotropic 2D convolutions.

Three other lines supply the method. First, Depth Coefficients for Depth Completion shows that a squared-error loss rewards predicting the average of foreground and background at a boundary. It replaces that loss with cross-entropy over soft two-hot depth bins, which reduced KITTI boundary error (tMAE) from 59.4 to 33.4. We reuse this bin representation as the indexing variable of our gate. Depth Completion Using Plane-Residual Representation adds a continuous residual on top of the discrete label. We adopt that residual to avoid staircase artifacts.

Second, Attention Gated Networks introduced additive gates in which coarse decoder features decide which encoder features pass through a skip connection. These gates have so far been formulated only for isotropic image grids. Our gate keeps their "decoder decides what passes" logic but replaces the 2D spatial map with a distribution over range.

Third, the audio literature constrains how the frequency side should be built. DTF-AT found that moderate time–frequency decoupling (6×3 kernels) beat extreme decoupling (18×1). Golden Gemini is All You Need shows that symmetric downsampling damages acoustic features because the two axes carry different physics. Towards Learning Universal Audio Representations argues that square kernels smear fine acoustic structure. We read these results as a warning against fully separating the axes. Our encoder therefore keeps a small amount of time context inside each delay window instead of using pure K×1 filters.

One constraint cuts against us. BatVision and the distillation study both argue that audible sound has low spatial bandwidth, so some blur of thin obstacles may be physically unavoidable. If so, forcing a discrete commitment could produce confident mistakes. We treat this as a hypothesis to measure, not one to argue away.

Mechanism

We first cross-correlate each recorded channel with the emitted chirp. This yields binaural impulse responses whose time axis is round-trip delay. We then compute their spectrograms, together with interaural level and phase difference channels.

The audio encoder never downsamples along time below the depth-bin resolution. It applies stacks of mildly anisotropic kernels that are tall in frequency and narrow in time, and it pools frequency into a feature vector. The output is one feature $a_k \in \mathbb{R}^C$ per delay window $k = 1,\dots,K$. Window $k$ covers delays $[2d_k/c,\ 2d_{k+1}/c)$ for depth-bin edges $d_k$. On BV1, with depth clipped at 12 m, this spans roughly 70 ms.

At each decoder scale, pixel $u$ predicts bin probabilities $p_k(u)$ and a direction query $q(u)$. The audio skip feature delivered to that pixel is

$$ s(u) = \sum_{k=1}^{K} p_k(u)\,\sigma\!\big(w^\top \phi(q(u), b_k)\big)\, a_k , $$

where $b_k$ is the binaural-difference feature of window $k$, $\phi$ is an additive compatibility function in the style of Schlemper et al., and $\sigma$ is a sigmoid. The sigmoid term says whether the energy at that range arrived from the pixel's direction.

Bin probabilities from coarse scales gate the skips at finer scales, so evidence sharpens from coarse to fine. The final head outputs a two-hot bin distribution trained with cross-entropy, plus a per-pixel residual within the bin. In the audio-visual track, the decoder trunk is driven by RGB features, and audio enters only through $s(u)$. Here the image proposes where edges are, and the echo confirms at which ranges surfaces actually exist. This is the role AVS-Net leaves unfilled.

Research Program

The work has three parts. The first is building and validating the deconvolution front end, so that the time axis really is delay; we will check alignment against stereo depth on BV1.

The second is training audio-only and audio-visual variants on BatVision BV1, with BV2 used for cross-site generalization. Protocols follow the BatVision dataset paper and AVS-Net, and we will check which train/test splits each prior paper used before comparing numbers.

The third is a boundary analysis that standard depth metrics miss. BatVision reports no edge metric, so we define three from the ZED ground truth. The first is a boundary band, consisting of pixels within a fixed distance of depth discontinuities above a relative jump threshold. The second is a ghost-pixel rate: the fraction of band pixels whose prediction lies strictly between the local foreground and background depths by more than 10% of the gap. The third is boundary RMSE and MAE in the spirit of KITTI's tMAE. We also stratify results by object width and by the delay separation between foreground and background, because the mechanism predicts gains only when the two surfaces fall in different windows.

Baselines

The naive reference is a per-pixel mean-depth predictor, for which BatVision reports an L1 of 0.10 on a 0–1 scale against roughly 0.08 for its model. Any variant that fails to clear this gap in the boundary band is not extracting local structure.

The direct architectural reference is the audio-only U-Net from The Audio-Visual BatVision Dataset, which reports approximately 1.33 m RMSE on BV1. We retrain it in our pipeline with the same data, augmentation, and epochs, so that numbers are comparable and do not depend on reproducing their exact preprocessing.

The strongest existing audio-only method is V2E-CCL from Evaluating Visual-to-Echo Distillation. It reports δ<1.25 of 0.569 against BatNet's 0.480, and a 15.8% relative REL improvement over SAGENet. We run it with its frozen visual teachers and give it the same tuning budget we give our own model, about twenty configurations each. Beating it means higher δ<1.25 on the same split together with a lower ghost-pixel rate, with seed variance reported. Matching global δ while clearly cutting boundary error would still support the core claim, and we would describe it that way.

In the audio-visual track, the strongest competitor is AVS-Net. Its reported number is the zero-shot BV2→BV1 setting, where Monodepth2's Abs Rel drops from 0.649 to 0.289. We reproduce that protocol and compare at the same visual backbone. AVS-Net applies a global scale only, so it should match us on global metrics after scaling and lose in the boundary band.

Four ablations isolate the key ingredient, all at matched parameter count and compute. The first keeps the BatVision U-Net and changes only the loss to Depth Coefficients plus residual. The second keeps L1 but uses our decoupled encoder. The third replaces our range-tied gate with unconstrained multi-head cross-attention from decoder pixels to all audio time–frequency tokens, using learned positional encodings. The fourth is our full model with the bin-to-delay assignment randomly permuted, which keeps the architecture identical while destroying the physics.

Why not just…?

Why not just swap the L1 loss for Depth Coefficients and keep the BatVision U-Net? This is the cheapest serious competitor. The KITTI result in Depth Coefficients for Depth Completion shows that the loss alone removes much of the averaging behavior. The weakness is that a latent-vector decoder has no per-range evidence to decide which mode a boundary pixel belongs to. It can only become confidently wrong, which is the risk raised by the low-bandwidth argument in the BatVision and distillation papers. The first ablation measures this directly. If loss-swap alone captures most of the boundary gain, the gate is not justified and we will report that outcome.

Why not just use a decoupled audio encoder such as DTF-AT and keep everything else? DTF-AT shows that decoupling helps audio classification, with 0.486 mAP on AudioSet at about a third of AST's compute. But classification pools time away, and a better encoder feeding a single bottleneck still loses the range axis. The second ablation tests whether encoder quality alone moves boundary metrics. We expect it to help global REL but not the ghost-pixel rate.

Why not just let generic cross-attention learn the time-to-range mapping? This is the real threat. With enough data, cross-attention with positional encodings could discover the delay–range correspondence unaided, and AVS-Net already uses cross-modal attention successfully. Our bet is that BatVision is small (52k BV1 and 3k BV2 instances), so the hard-wired prior should win on data efficiency and on cross-site transfer to BV2. We will test this with the third ablation across training-set fractions of 10%, 30%, and 100%. If unconstrained attention matches us at full data, the honest conclusion is that the prior mainly buys sample efficiency.

Why not just use audio for global scale, as AVS-Net does? Global scale cannot move a boundary pixel from one surface to another, as AVS-Net's authors themselves state. The audio-visual comparison restricted to the boundary band, and to scenes with glass or specular surfaces where RGB edges mislead, will show whether dense range-gated audio adds anything beyond scale.

Decisive Checks

Four results would settle the proposal. The first and most important is the permuted-delay control. If scrambling the bin-to-delay assignment costs little, the gain comes from capacity rather than physics, and the core claim fails.

The second is the stratified boundary analysis. Gains should grow with the delay separation between foreground and background, and should be near zero when both surfaces share a window. A flat profile would contradict the proposed mechanism.

The third addresses the bandwidth tension directly. For boundary pixels where the model commits, meaning bin entropy below a threshold, we measure the accuracy of the chosen mode as a function of object width. If committed predictions on thin objects are mostly wrong, discrete commitment is harmful at audible wavelengths. In that case we would switch the head to report calibrated bimodal output rather than a single depth.

The fourth is cross-site transfer from BV1 to BV2 and back. A physically grounded gate should degrade less than unconstrained attention when room acoustics change.