Keeping Two Answers Alive: Multi-Peak Cascaded Stereo for Full-Resolution Transparent Scenes

Stereo depth networks save memory by working coarse-to-fine. They first make a rough depth guess for each pixel at low resolution, then search only a narrow depth window around that guess at higher resolution. This works when each pixel has one true depth. A pane of glass or a beaker breaks it. Looking through glass, a camera sees two plausible matches along the same ray: the glass surface and the background behind it. The coarse stage tends to lock onto the high-contrast background. The fine stages then search only near that wrong answer and can never recover the glass.

We propose a cascaded stereo network that keeps every strong peak of the coarse matching distribution, not just one. Each surviving peak gets its own small high-resolution search window. A fixed memory budget is spent where the scene is actually ambiguous: one window for ordinary opaque pixels, several for transparent ones. The goal is full 12-megapixel stereo on glass-heavy scenes without downsampling, producing layered depth (the glass and what lies behind it) at a memory cost close to a standard cascade.

Core Claim

The memory-saving step in modern stereo and multi-view stereo, narrowing the search around one coarse estimate, is the specific reason these methods fail permanently on transparent surfaces. The fix is to make the narrowing multi-modal and to allocate the hypothesis budget adaptively, not to abandon cascades.

Two lines of work motivate this.

  • The narrowing step and its assumption. Cascade Cost Volume builds successive stages that narrow the depth search around the previous prediction. It halves memory on DTU (10.8 GB to 5.3 GB). IS-MVSNet goes further and places depth samples geometrically around the coarse estimate. It justifies this explicitly by assuming errors are unimodal and zero-centred.
  • Transparency violates that assumption. Multi-Label Stereo Matching for Transparent Scene Depth Estimation shows that transparent pixels carry two disparities. It regresses a two-component Gaussian (foreground, background, and their correlation) with recurrent updates.

That paper is the closest existing work, and our proposal differs from it in three ways. It fixes the number of layers at two, while ours varies per pixel. It refines regression parameters, while we place explicit, budgeted cost-volume windows. It does not address multi-megapixel memory, which is our central constraint.

Open Challenges in Deep Stereo: the Booster Dataset supplies the setting. It is a set of 12 Mpx stereo pairs with glass and mirrors. Its authors report that volumetric networks must downsample inputs by four to fit in GPU memory, and that bad-2 error (the fraction of pixels more than 2 px wrong) exceeds 40% overall and often exceeds 80% on transparent and specular surfaces.

Research Program

The network has three stages.

Coarse stage. A cost volume at 1/8 resolution spans the full disparity range. Each pixel gets a probability distribution over disparity, which we call its matching profile.

Peak extraction. Instead of taking the expected value, we extract up to $K_{max}$ local maxima whose probability mass exceeds a learned threshold.

Budgeted fine stages. A tile-level allocator distributes a fixed budget of fine disparity planes across the extracted peaks. It allocates in proportion to each peak's mass and to the entropy of the coarse profile. Fine stages then build small windows of $d$ planes around each kept peak, stored as $K_{max}$ padded slots with unused slots masked. They regularise these windows with the same 3D convolutions a standard cascade uses.

With feature channels $C$, image size $H \times W$, full disparity range $D$, coarse downsampling factor $s$, and $\bar{K}$ the average number of kept peaks per pixel, the dominant memory cost scales as

$$M \propto C\,H\,W\left(\frac{D}{s^{3}} + \bar{K}\,d\right).$$

The first term is the coarse volume: downsampling divides $H$, $W$ and $D$ each by $s$. The second term is the fine windows. Opaque-dominated scenes keep $\bar{K}$ near one, so memory stays close to a standard cascade. Transparent regions are usually a minority of pixels, so the extra cost is bounded by how much of the image they cover.

Output and training. The output is an ordered set of depth layers with confidences. Training uses a set loss: each ground-truth layer is matched to its closest predicted layer, and unmatched predictions are pushed toward low confidence.

  • Two-layer supervision. This comes from TranScene, the synthetic multi-label set of Multi-Label Stereo Matching.
  • Booster supervision. Booster's ground truth was captured on painted scenes, so it gives the front surface only. We supervise the nearest confident layer against it.

We also ablate one secondary component, the final upsampling guidance. StereoNet refines a coarse disparity using the full-resolution RGB image as an edge guide. ClearGrasp argues that colour edges on glass are unreliable, because they come from refracted background, caustics, and contact edges rather than geometry. We therefore compare RGB guidance against guidance from per-layer cost features.

Baselines

Naive reference. A standard unimodal cascade, Cascade Cost Volume applied to stereo, run on Booster at 1/4 resolution with the output upsampled. This is what current practice does, according to Booster. The same cascade is also run at full resolution under our memory budget, to separate the effect of resolution from the effect of multiple peaks.

Strongest single-layer competitor on Booster. Fine-tuned RAFT-Stereo. Booster reports that fine-tuning reduces its MAE from 17.08 px to 7.56 px. It is the strongest baseline because iterative correlation already avoids dense 3D volumes. We retrain it on the same Booster split with the same GPU-hour budget.

Strongest layered competitor. The two-layer Gaussian method of Multi-Label Stereo Matching. It reports TranScene MAE of 0.014 m against 0.188 m for RAFT-Stereo. No reported Booster result for it was found on Lacuna, so we run it ourselves at the largest resolution that fits the shared memory cap.

Criteria for beating them.

  • On Booster, beating the competitors means lower bad-2 and MAE on transparent-mask pixels at equal or lower peak memory, with no regression on opaque pixels.
  • On TranScene, it means matching 0.014 m on the foreground layer while also recovering the background layer.

Key ablations.

  • Matched-memory widening. The same total number of fine planes is spent as one wide window per pixel instead of several narrow ones.
  • Fixed versus adaptive budget. A fixed $K=2$ for every pixel replaces the adaptive allocator.

Why not just…?

Why not just widen the fine-stage search window? This is the most serious threat. If one wide window covering both glass and background works as well, the multi-peak machinery is unnecessary. The matched-memory widening ablation tests exactly this. Glass and background are often separated by large disparity gaps at 12 Mpx, so a single window wide enough to cover both would have to space its planes coarsely, giving up the fine sub-pixel sampling that IS-MVSNet shows matters. If the wide window ties on Booster transparent pixels, we will report that and drop the claim.

Why not just run a recurrent correlation model like RAFT-Stereo, or the two-layer Gaussian method, which already avoid 4D volumes? Both remain single-scale iterative models. Booster's authors still found them memory-limited at 12 Mpx. The two-layer method also cannot represent nested transparency of the kind catalogued in ClearPose, such as items inside translucent covers. Our head-to-head under a shared memory cap settles which is better at full resolution.

Why not just opaquify the images with GHOST and run a normal cascade? GHOST paints opaque textures over glass so that downstream reconstruction models work better. It raises DUSt3R F-score from 77.36 to 88.66. However, it costs 0.38 s per view on an RTX 4090, relies on a weak-refraction assumption, and erases the background layer by design. It also synthesises each view separately and may not stay consistent across views, which would hurt stereo matching. We include GHOST followed by a cascade on Booster to test this directly.

Why not just downsample and refine with RGB guidance, as StereoNet does? This is the 1/4-resolution reference above. The guidance ablation tests whether colour edges actually mislead the refinement on refractive boundaries.

Proposed Experiments

Go/no-go check: peak recall. The first experiment is a cheap diagnostic that decides whether the idea can work at all. On Booster transparent masks and TranScene, we measure how often the 1/8-resolution matching profile contains any peak within 2 px of the true foreground disparity. If the coarse stage never sees the glass, no amount of fine budgeting helps. In that case the project pivots to raising coarse resolution only inside transparency masks.

Main comparison. On Booster at full resolution we report:

  • bad-2 and MAE, split by Booster's material masks;
  • peak GPU memory and latency;
  • the same metrics on opaque pixels, to check for regression.

On TranScene we report per-layer MAE.

Generalisation to multi-view stereo. Porting the peak-extraction stage into a multi-view cascade and evaluating on DTU should preserve CasMVSNet-level accuracy and memory, since opaque scenes keep $\bar{K}$ near one.

Decisive outcome. The idea is confirmed if two things hold. First, at matched memory, multi-peak beats widened-window on transparent pixels. Second, the adaptive allocator beats fixed $K=2$ in accuracy per gigabyte.