Phones and robots often pair a sharp main camera with a much cheaper second camera. Stereo depth, which computes how far each pixel shifts between the two views, then has to match fine detail in one image against a blurry, low-resolution version in the other. Methods built for this "unbalanced" setting all work on single frozen image pairs. Methods for video depth in moving scenes all assume two identical cameras. This proposal argues that the gap between them is the opportunity. When a camera moves through a mostly static world, a low-resolution sensor samples each surface at slightly different sub-pixel positions over successive frames. Accumulated over time, those samples can recover the fine detail that any single frame lacks. The catch is that this works only where the scene is actually static. On moving people or objects, the same accumulation smears and invents detail.
We propose a continuous spacetime disparity model for unbalanced stereo video. It is a small network queried at any pixel location and time. Its inputs are encoded with sinusoidal features, which let the network represent sharp changes. The key design choice is that the frequency range the network may use is set per location by how much evidence exists there. Static regions seen across many frames get high-frequency encodings and inherit detail gathered over time. Moving regions are limited to what a single low-resolution frame can support. The decision about what is moving is made inside the high-resolution stream alone, comparing it with itself over time. It never relies on comparing the two mismatched cameras with each other, which is the comparison that unbalanced optics corrupts.
Core Claim
In unbalanced stereo video, the frequency content a disparity map can legitimately contain varies across the image. It depends on two things: the secondary sensor's resolution, and the number of frames over which a point has stayed static and visible. A model whose sinusoidal encoding bandwidth is gated by this observable bandwidth will beat per-frame continuous refinement on static regions, because it uses time. It will also beat ungated high-frequency encodings on dynamic regions, because it does not hallucinate detail there. The specific new elements are three: a bandwidth gate driven by evidence, applied to a coordinate network for disparity; temporal accumulation from the low-resolution view; and motion detection that is decoupled from cross-view photometric residuals.
Where This Sits
Two works supply the method's starting points. Neural Disparity Refinement for Arbitrary Resolution Stereo refines a classical Semi-Global Matching disparity map, called the SGM seed, using multi-layer perceptrons (MLPs). The MLPs are queried at continuous coordinates through bilinearly interpolated features, and the method handles secondary views up to 12x lower in resolution. It uses no sinusoidal encoding and sees one static pair. We keep its classification-plus-offset output head and its continuous queries, and we add time and gated encodings. The sinusoidal mapping itself comes from Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. That paper shows that coordinate networks learn smooth functions first (spectral bias) and that projecting coordinates through sines and cosines widens what they can fit. D-NeRF already encodes space and time this way for dynamic radiance fields. It assumes identical, clean views and uses no stereo geometry, whereas we put the encoding on a disparity field driven by a stereo pair of unequal quality.
Two works motivate the problem. Visually Imbalanced Stereo Matching shows that standard deep stereo networks "collapse" past 5x–8x degradation, with error above 80%. Its restore-then-match approach holds about 22% error at 20x blur, but only on static pairs. DynamicStereo obtains temporally consistent video depth with attention across space, stereo and time. It needs matched cameras, because its correlation volume and attention tokens assume equal resolutions.
Two works constrain the design. Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction finds that drift in the high-frequency slope of features correlates negatively with 3D quality, so injecting high frequencies everywhere is actively harmful. This is the reason our encoding is gated rather than simply widened. Robust distractor methods, RobustNeRF (trimmed photometric loss) and NeRF On-the-go (uncertainty predicted from DINOv2 features with an SSIM loss), flag moving content by its high photometric residuals. Under unbalanced optics, residuals between the two views are high everywhere. So we take only the idea of robust outlier handling from them and move the residual test into the high-resolution stream over time.
Research Program
The model takes a sliding window of $T$ frames from a high-resolution primary camera and a low-resolution secondary camera. Its first stage is a dynamics estimate computed only on the primary stream. Primary frames are warped to one another using camera ego-motion and optical flow. Pixels whose warped appearance is inconsistent, measured with a structural similarity residual following NeRF On-the-go, are marked as possibly dynamic. The output is a per-pixel count $n_{\text{eff}}(\mathbf{x},t)$: the number of window frames in which that surface point was static and visible.
The second stage is temporal gathering. For each query point, features from the secondary view are sampled at every frame where the point counts as static, at the location implied by the current disparity hypothesis and the camera motion. Because the camera moves, these samples land at different sub-pixel phases of the secondary sensor grid. A small attention pool combines them.
The third stage is a gated spacetime query network. Following Aleotti et al., the MLP predicts a disparity bin and a sub-pixel offset. Its input includes a sinusoidal encoding of $(x,y,t)$ in which each frequency band $\mathbf{b}_k$ is weighted by
$$w_k(\mathbf{x},t)=\sigma\!\left(\frac{\beta(\mathbf{x},t)-\lVert\mathbf{b}_k\rVert}{\tau}\right),\qquad \beta(\mathbf{x},t)=f_N\cdot\min\!\left(s,\sqrt{n_{\text{eff}}(\mathbf{x},t)}\right).$$
Here $\sigma$ is the logistic function, $\tau$ is a softness temperature, $f_N$ is the Nyquist frequency of the secondary sensor, and $s$ is the resolution ratio between the two cameras. The square root is a deliberately conservative heuristic, borrowed from noise averaging in multi-frame super-resolution: more frames add evidence with diminishing returns, and the gain is capped at the true resolution ratio. We will compare this hand-set schedule against a version in which $\beta$ is predicted by a small network from $n_{\text{eff}}$ and local primary-image gradients.
Initialization swaps the SGM seed for a coarse disparity map from a low-resolution correlation volume. We also test SGM, to measure how much the seed limits accuracy, which is a known weakness of Aleotti et al. Training is supervised on Dynamic Replica and SceneFlow. Unbalance is simulated by downsampling the right view by 4x, 8x and 12x, then adding independent blur, noise and compression, following the degradation recipes of Zhou et al..
Baselines
The naive reference is bicubic upsampling of the secondary view to the primary's resolution, followed by off-the-shelf DynamicStereo, run on identical degraded windows. VIS reports collapse beyond 5x–8x for comparable per-frame pipelines. We expect this baseline to fail at 8x and 12x, but at 4x it may be competitive, and we will report that honestly.
The strongest per-frame competitor is Aleotti et al.'s neural disparity refinement, run frame by frame and followed by a temporal median filter over flow-aligned frames. It is the strongest because it already handles 12x unbalance and isolates exactly what time adds. It will be retrained on the same degraded Dynamic Replica plus SceneFlow data, with the same seed and the same number of GPU-hours. The file gives no exact decimal results for it (no reported result found on Lacuna for its UnrealStereo4K or KITTI end-point error).
The strongest temporal competitor is DynamicStereo retrained end-to-end on the same asymmetric data, with the secondary view upsampled. It is a real threat because its attention across time could learn to accumulate detail implicitly. Its reported symmetric figure is a Sintel Clean temporal end-point error (TEPE, which measures frame-to-frame depth flicker) of 0.779, against 0.920 for RAFT-Stereo (DynamicStereo). The asymmetric setting has no reported number on Lacuna. For static, single-frame reference points we will include VIS and SASS. SASS reports an end-point error of 2.183 at 4x downsampling on KITTI 2015, against 2.652 for its baseline (Unsupervised Deep Asymmetric Stereo Matching with Spatially-Adaptive Self-Similarity). We will match that protocol on its single-frame track.
Beating the competitors means lower end-point error on static pixels than per-frame refinement with smoothing at 8x and 12x. It also means no worse end-point error and lower TEPE on dynamic pixels than retrained DynamicStereo, using equal parameter and compute budgets and the same hyperparameter-search budget for every method.
Four ablations isolate the key ingredient. The first removes the sinusoidal encoding entirely, giving bilinear features only. The second uses an ungated, full-bandwidth encoding. The third keeps the gate but sets $n_{\text{eff}}$ from cross-view residuals instead of the primary stream's temporal consistency. The fourth uses $T=1$. The second and third ablations are the ones that test the claim.
Why not just…?
Why not just add positional encoding to Aleotti et al.'s MLP? That is the first two ablations: no encoding, and fixed full bandwidth. The spectral-probing results warn that indiscriminate high frequencies hurt geometric consistency. A single frame also lacks the evidence to support detail beyond the secondary sensor's Nyquist limit. If the ungated variant matches the gated one on dynamic pixels, the gate is unnecessary, and we will report that.
Why not just run per-frame unbalanced refinement and smooth over time? Smoothing reduces flicker but cannot add detail. Every frame's estimate is capped by the same low-resolution evidence, and median filtering lowers resolution further at motion boundaries. The decisive test is whether static-pixel error keeps falling as the window grows. Under per-frame refinement with smoothing it should plateau. Under gathering it should not.
Why not just retrain DynamicStereo on unbalanced data? This is the most serious alternative. Its attention may learn accumulation by itself, but it has no explicit notion of which pixels can be accumulated. It also cannot use dynamics evidence computed from the primary stream alone. We settle this with equal-budget training. We report static and dynamic pixels separately, and separately for each degradation factor, so that a tie on static regions combined with a loss on dynamic ones (or the reverse) is visible rather than averaged away.
Why not just mask moving objects with a video segmenter? Split4D reports that pipelines built on external trackers drift. Its tracking-free method reaches 0.893 mIoU against 0.592 for SA4D on Multi-Human. RobustNeRF notes that class-based segmenters also miss unmodeled distractors such as shadows. We will nonetheless include a variant that sets $n_{\text{eff}}$ from an off-the-shelf segmenter.
Proposed Experiments and Decisive Checks
The main benchmark is Dynamic Replica and Sintel (Clean and Final) with synthetic 4x, 8x and 12x secondary-view degradation. Metrics are end-point error, the bad-2 fraction (pixels with error above 2), and TEPE, each reported on static and dynamic masks taken from the ground-truth segmentation. We also run KITTI 2015 under the SASS protocol, and the Booster 12 Mpx versus 1.1 Mpx unbalanced pairs as a real-sensor static test. On Booster, published models exceed 40% bad-2 overall and 80% on specular or transparent surfaces. We report per material class, since view-dependent highlights may wrongly be flagged as dynamic.
There are three decisive checks. The first is a window-length sweep ($T$ from 1 to 20) on static pixels: error should fall as $T$ grows and saturate near the resolution ratio. The second applies the spectral-probing diagnostics, high-frequency slope drift and structural spectral consistency, to disparity maps on dynamic pixels, where the gated model should show less slope drift than the ungated one. The third swaps the cross-view-residual gate for the primary-stream gate. The primary-stream gate should mark far fewer static pixels as dynamic at 12x, because cross-view residuals are inflated everywhere by the imbalance itself. If all three hold, the main claim holds: time supplies resolution the second camera lacks, but only where the scene is genuinely static.
References
- Neural Disparity Refinement for Arbitrary Resolution Stereo
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- D-NeRF: Neural Radiance Fields for Dynamic Scenes
- Visually Imbalanced Stereo Matching
- DynamicStereo: Consistent Dynamic Depth from Stereo Videos
- Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- Towards real world stereo image super-resolution via hybrid degradation model and discriminator for implied stereo image information
- Unsupervised Deep Asymmetric Stereo Matching with Spatially-Adaptive Self-Similarity
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Open Challenges in Deep Stereo: the Booster Dataset