Rate-Limited Cost Volumes for Synthetic-to-Real Stereo

Stereo depth networks usually learn from synthetic image pairs, where exact depth is free, and are then used on real cameras, where it is not. Their central data structure is the cost volume: a 4D grid that scores how well each pixel in the left image matches each candidate shifted pixel in the right image. Its size grows with height, width, number of candidate shifts, and feature channels, so it is usually the memory bottleneck. We propose to pass this volume through a learned quantizer with a bit budget. The quantizer rounds each entry to an integer and charges the network for every bit it keeps. Rounding has zero gradient almost everywhere, so during training we replace it with the standard stand-in from learned image compression: add uniform noise of width one. A single rate-penalty weight then controls how much information the volume may carry.

The bet is that one mechanism does two jobs. A volume stored at a few bits per entry needs a fraction of the memory of a 32-bit float volume. That saving can be spent on finer spatial resolution, which matters for thin structures. Separately, a network that cannot afford to carry rendering-specific detail through its matching stage should transfer better from synthetic to real images. The claim is falsifiable. If the bottleneck helps generalization, zero-shot real-world error should be U-shaped in the bit rate, while synthetic error rises steadily as the budget shrinks. If only memory improves, the idea reduces to activation compression, which is still useful but much less interesting.

Core Claim

Training with an entropy-coded, rounded cost volume gives lower zero-shot error on real driving stereo than three alternatives: an unconstrained float volume, a volume of equal memory made smaller by cutting channels, and a volume that gets the same training noise without any rate penalty. At equal memory, the saved bits let the volume run at twice the spatial resolution, which reduces errors on thin structures.

Where This Sits in Prior Work

The quantization machinery is taken from learned image compression. End-to-end optimization of nonlinear transform codes for perceptual quality and Nonlinear Transform Coding train through rounding by adding uniform noise, then round for real at test time. The noisy values have a density that matches the histogram of rounded values, so a learned probability model can estimate the true bit cost during training. We reuse this unchanged. The difference is where we apply it: to an intermediate matching representation judged by depth error, not to an image judged by reconstruction.

The memory problem itself is well documented. Kendall et al. introduced concatenation cost volumes smoothed by 3D convolutions. Chang et al. documented the resulting memory growth. StereoNet avoids that growth by building the volume at 1/8 or 1/16 resolution, then restoring edges with refinement guided by the color image. It still stores every entry as a 32-bit float. The file also notes that its color-guided refinement leaks wrong boundaries when image edges and depth edges disagree. That paper supplies our backbone. Group-wise correlation (Xu, CIDER) roughly halves volume memory by collapsing channels into similarity scores. It is the capacity-reduction competitor our proposal must beat.

Motivation for the generalization half comes from outside stereo. Deep Learning in Medical Image Registration: Magic or Mirage? shows that deep registration models beat classical methods on their training distribution but lose to them on external scanners, because they memorize training-set structure. The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions finds that 32 of 48 studies train only on synthetic X-ray projections and describes the resulting "synthetic trap." MRIComp4Flow reports that mild offline JPEG2000 compression of MRI acts as a denoising regularizer. It also reports that heavier compression erodes small structures, a warning that the rate knob has a real downside. An Information-Theoretic Regularizer for Lossy Neural Image Compression supports treating the rate term as an information limit.

None of these works places a trainable rate constraint on a cost volume, and none measures synthetic-to-real transfer as a function of bit rate. MRIComp4Flow comes closest in spirit. However, its codec is fixed and sits outside training, so the network cannot choose which information to discard. Continuous Patch Stitching combines learned compression with high-resolution memory limits, but only for 2D images on clean benchmarks.

Research Program

Backbone. We use a StereoNet-style network. Shared 2D features from both images are combined into a 4D volume. A 1×1×1 3D convolution maps each entry to $C$ channels, giving the latent volume $y$. During training we use $\tilde y = y + u$, with $u$ drawn uniformly from $(-0.5, 0.5)$. At test time we use the rounded volume $\hat y = \lfloor y \rceil$. A small per-channel factorized entropy model $p$ gives the rate. The 3D smoothing network, the soft-argmin that turns match scores into a disparity, and the color-guided refinement are all unchanged. Training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{disp}}(d, d^{*}) + \lambda \, \frac{1}{|V|}\sum_{v \in V} -\log_2 p(\tilde y_v),$$

where $d$ is the predicted disparity, $d^{*}$ the synthetic ground truth, $V$ the set of volume entries, and $\lambda$ the price of one bit per entry. We sweep $\lambda$ to trace a curve of rate against error.

Memory. Rounded codes can be stored as low-bit integers and decoded on the fly when the smoothing network needs them. The same trick applies during backpropagation, where decoding replaces the cached float activations, as in the gradient checkpointing of Blumberg et al..

We are careful about the size of this saving. The float activations inside the 3D smoothing layers remain, and the compression does not remove them. We will report stored-volume memory and total peak memory separately.

The equal-memory high-resolution variant builds the volume at 1/4 resolution instead of 1/8. That has eight times as many entries: twice the height, twice the width, and twice the candidate shifts. At roughly 4 bits per entry against 32, the stored volume is the same size.

Analysis. For each $\lambda$ we record synthetic error, zero-shot real error, and realized bits per entry. We also measure which information the bottleneck removes. The test is to correlate per-entry rate with image properties that differ between rendering and real cameras, such as texture energy and saturated regions.

Baselines

The naive reference is the same backbone trained identically on Scene Flow with a 32-bit float volume and no rate term. It is evaluated zero-shot on KITTI 2012 and 2015. Every other result is reported as a change from it.

The strongest existing method in the file is StereoNet, which reports an end-point error of 1.1 on Scene Flow. That figure lets us confirm our reimplementation is faithful. For zero-shot KITTI, the file gives no number: no reported result found on Lacuna for StereoNet D1-all on KITTI 2012/2015. We will therefore run it ourselves.

The strongest direct competitor on memory is a group-wise-correlation volume with its channel count set so that its stored memory equals our quantized volume. It tests whether any compact volume would do as well. Post-training quantization is also included: the naive float model with its volume rounded to 8 and 4 bits after training, with no retraining. It isolates the value of the network adapting to the quantizer.

The key ablation is noise-only. It uses identical uniform noise and rounding at test time but sets $\lambda = 0$, so there is no pressure on rate. If this matches the full method, then the regularizer is the noise and not the bit budget. A second ablation swaps the noise stand-in for straight-through rounding, which rounds in the forward pass and passes gradients unchanged. It checks sensitivity to the mismatch between training and test.

Fairness. All runs share the same data, the same schedule, and the same color and contrast augmentation. Each gets an equal tuning budget of eight configurations. Checkpoints and hyperparameters are chosen on held-out Scene Flow only, never on KITTI.

Our operating $\lambda$ is chosen before any KITTI evaluation, at the knee of the synthetic rate–error curve. This removes the tempting shortcut of choosing the rate on real data.

What counts as beating. The full method wins if, at the pre-registered $\lambda$, zero-shot KITTI D1-all is lower than both the float reference and the equal-memory group-wise model. The gap must hold across three seeds. Synthetic error may be equal or somewhat worse.

Why not just…?

Why not just downsample the volume further? Downsampling saves memory by discarding spatial detail uniformly. The file shows that thin and small structures suffer first. MRSegmentator reports thin vessels near 0.64 Dice against 0.96 for the lungs. The equal-memory comparison between a 1/4-resolution quantized volume and a 1/8-resolution float volume, scored on thin-structure and disparity-edge pixels, is our direct test of this.

Why not just quantize after training? Offline quantization gives the memory saving without letting the network decide what to keep. MRIComp4Flow's fixed codec loses small tumor structure at 50:1 and 100:1 compression. We expect the same thing to happen to thin disparity structure at 4 bits or fewer. The post-training quantization baseline measures this directly.

Why not just use fewer channels or group-wise correlation? This is a real threat. A narrow float volume is also a bottleneck, and it is simpler. The difference is that a channel cap limits capacity uniformly, whereas a rate penalty lets the network spend bits where matching is ambiguous and spend almost nothing in easy regions. If the equal-memory group-wise baseline matches us on KITTI, we will report that bit budgets add nothing beyond width.

Why not just add noise or dropout to the volume? This is the most likely way the idea fails. The training stand-in for rounding already injects noise. The noise-only ablation settles the question. If it closes most of the gap, the contribution shrinks to a memory result plus a finding about noise regularization, and we will say so.

Decisive Checks

The central experiment is the $\lambda$ sweep on Scene Flow with zero-shot evaluation on KITTI 2012 and 2015 at each rate. A U-shape on real data, alongside steadily rising synthetic error, supports the claim. A curve that stays flat or falls steadily refutes it.

The idea survives only if two gaps are larger than seed variance at matched memory. The first is between the full method and noise-only. The second is between the full method and the equal-memory group-wise model.

A secondary check is qualitative. On KITTI regions where color edges and depth edges disagree, which is StereoNet's documented bleeding failure, we test whether the rate-limited model keeps its error concentrated at true depth boundaries.

As a stretch goal, the same bottleneck placed on deformation features could be tested on the OASIS-to-external-cohort split from the Mirage study. That split has a published pattern of deep models losing out-of-distribution. We treat it as an extension, not a precondition.