Keeping the Second Minimum: Memory-Bounded Cost Volumes for Transparent Objects

Multi-view depth networks estimate depth by testing a stack of candidate depths for every pixel and checking which candidate makes the views agree. That stack, called a cost volume, grows with image size times the number of candidates, so it quickly runs out of GPU memory. The standard fix is coarse-to-fine search: test few candidates at low resolution, keep a narrow window around the best one, and refine. On glass this fix fails in a specific way. Looking through a clear vial, the views agree most strongly at the depth of the table behind it, because the background texture is what the cameras actually see. The glass surface itself produces only a weaker, secondary agreement from rims, highlights and dust. Coarse-to-fine search throws that secondary agreement away at the first stage. Plain argmin over a full-resolution volume would discard it too.

We propose that transparent surfaces in a cost volume behave like the transparent surfaces that α-NeuS found in neural distance fields: they are local minima, not the global minimum. A memory-bounded cost volume should therefore keep several depth modes per pixel instead of one, and choose among them with cheap transparency cues. A by-product is a two-layer depth map, giving the first surface and the surface seen through it, which is the structure that breaks current methods on nested transparency. Every comparison is run under matched peak-memory and latency caps. This borrows the lesson of Benchmarks as Software: when compute budgets are left unmatched, leaderboard gaps can reflect budget rather than capability.

Core Claim

The failure of cost-volume stereo on transparent objects comes from two sources, and they can be separated. One is the photo-consistency violation itself. The other is how memory-saving search and single-mode depth readout handle it. Coarse-to-fine pyramids and soft-argmin regression both commit to one hypothesis per pixel, so they keep the background match or average glass and background into a floating depth.

We claim that a cost volume can do better at the same memory as a single-window pyramid. It should spend its fixed hypothesis budget on $K$ narrow windows around the $K$ strongest local minima of the coarse matching cost, rather than one wide window around the global minimum. A light selector conditioned on transparency segmentation and surface normals then picks the first-surface mode. Under that design, it should recover glass depth that both pyramids and dense argmin miss, at a fraction of dense-volume memory.

Memory makes the trade explicit. A dense volume costs $H W D C$ floats, where $H \times W$ is resolution, $D$ is the number of depth candidates and $C$ is feature channels. The fine stage of our design costs

$$M = H W C \sum_{k=1}^{K} b_k, \qquad \sum_{k} b_k = B \ll D,$$

where $b_k$ is the number of candidates in window $k$ and $B$ is the same total budget a single-window pyramid would use. The proposal changes where candidates go, not how many there are.

Prior Work and Where This Sits

Three works motivate the proposal.

α-NeuS showed that per-scene neural surface models already encode thin transparent surfaces as non-negative local minima of the distance field. Taking the absolute value of the field recovers them, cutting synthetic Chamfer distance from 23.41e-3 to 5.67e-3. We transport that observation from slow per-scene optimization to a feed-forward cost volume. We inherit its thin-wall assumption and will report thick-glass failures rather than hide them.

StereoNet and MVSNet supply the method we modify:

  • StereoNet uses a coarse volume plus RGB-guided edge refinement, reaching 15 ms at 720p, and names reflective glass as a failure case.
  • MVSNet uses frustum volumes with variance aggregation and soft-argmin readout.

Neither was evaluated on transparent data. We keep their warping and aggregation but replace single-mode readout and the raw-RGB refinement guide. ClearGrasp-style evidence suggests raw RGB edges on glass mark refracted background, not true geometry.

ClearPose motivates the two-layer output: pose ADD-S falls from about 58% to 13% under translucent covers.

Single-view transparent depth methods constrain the design and set the bar:

  • ClearGrasp uses normals and boundaries with global optimization.
  • TODE-Trans uses a transformer encoder.
  • FDCT runs at 70 FPS.

They work without parallax. ClearGrasp itself notes that regions enclosed by occlusion boundaries get indeterminate depth. Our selector reuses their idea that normals and segmentation are more reliable on glass than raw depth. It uses those cues only to choose among geometric modes, not to inpaint depth.

GHOST takes the opposite route: it paints transparent regions opaque so off-the-shelf models work, raising DUSt3R F-score from 77.36 to 88.66. It costs 0.38 s per view and hallucinates each view independently.

R3DP addresses latency by caching a heavy 3D foundation model. It reaches about 40.3 ms encoding, but builds no explicit depth-hypothesis search. We treat it as context for the latency target, not as a perception baseline.

Research Program

Diagnostic study. We first train a dense, memory-unconstrained MVSNet-style volume on ClearPose multi-view sequences. ClearPose camera poses come from its ORB-SLAM3-based labelling pipeline. Along each ray in labelled transparent regions, we measure "mode recall": how often the ground-truth glass depth lies within a tolerance of any of the top-$K$ local minima of the matching cost. We report it separately by ClearPose split. If glass depth is rarely any local minimum, the premise is false, and the project reduces to a negative result about photo-consistency on glass.

Multi-mode pyramid. We then build a pyramid whose coarse stage outputs, per pixel, the $K$ deepest local minima and their cost margins. The next stage allocates $b_k$ candidates to each window, in proportion to the margins, under a fixed total $B$. A small 2D head predicts transparency probability and normals from RGB. A selector then outputs two quantities: first-surface depth from the nearest mode that is both consistent with the predicted normals and significant, and through-surface depth from the remaining strongest mode. Refinement is guided by the predicted normal and segmentation maps instead of raw RGB edges.

Budget-matched protocol. All methods run under enforced peak-memory caps of 2, 4 and 8 GB and a latency cap matched to R3DP's roughly 40 ms observation budget. Each method gets the same tuning budget. We report accuracy-versus-budget curves and the Spearman rank correlation of methods across budgets.

Ground-truth audit. In the spirit of TB-Audit, we flag ClearPose frames whose CAD-rendered depth is inconsistent when reprojected across neighbouring views. We report results with and without flagged frames, so that annotation defects cannot masquerade as method differences.

Baselines

Naive references. These are raw sensor depth and the dense volume with soft-argmin readout. The dense volume is trained on the same ClearPose training splits at the largest resolution that fits each memory cap. It shows what one gets by doing nothing transparency-specific. Our method must beat it at every cap.

Single-window pyramid. This is a cost-volume-pyramid baseline in the style of CVP-MVSNet: coarse search, then one narrowed window around the prior estimate. It uses the same backbone, the same total hypothesis budget $B$ and the same training schedule. It is the closest architectural competitor, differing only in keeping one mode instead of $K$. Beating it at equal $B$ is the core claim.

Per-view transparent depth, fused. TODE-Trans and ClearGrasp run on every view and are fused with known extrinsics. These are the strongest existing transparent-depth methods. On single-view ClearGrasp real-known data, ClearGrasp reports RMSE 0.039 (ClearGrasp). TODE-Trans reports a 25% RMSE gain over prior convolutional baselines (TODE-Trans). No reported result found on Lacuna for multi-view fused depth on ClearPose, so we will produce it ourselves under the same splits.

GHOST preprocessing. GHOST is followed by DUSt3R and by our dense volume. The reported numbers are DA3 RMSE improving from 0.023 to 0.018 and DUSt3R F-score from 77.36 to 88.66 on ClearGrasp (GHOST). We report GHOST both with an unlimited latency budget and under the 40 ms cap.

Offline reference. α-NeuS on its own synthetic and real scenes serves as an accuracy ceiling for thin objects, not as a competitor.

Ablations. Three ablations isolate the key ingredient:

  • Dense volume with our local-minimum selector and no memory limit. This separates the readout rule from the sparse allocation.
  • $K=1$ with the transparency selector. This tests whether cues alone suffice.
  • $K$ modes without transparency cues, nearest-mode rule only. This tests whether the cues earn their cost.

What counts as winning. Beating the strongest baseline means lower first-surface depth error in transparent regions on ClearPose's translucent-cover and liquid splits than both the single-window pyramid and fused TODE-Trans, at equal memory and within the latency cap. It must do so without losing accuracy on opaque regions.

Why not just…?

Why not just run a transparent depth-completion network on each camera and fuse? This is the most serious threat on easy splits, where single-view cues may suffice. The evidence against it comes from the hard cases. ClearGrasp concedes indeterminate depth inside occlusion boundaries. ClearPose shows collapse under translucent covers, which is exactly where a second layer exists and a single-view network can output only one depth. The fused baseline above settles the question per split. If it matches us on translucent covers and liquids too, multi-view search is not worth its cost, and we will say so.

Why not just make the dense volume bigger, or use a larger GPU? If the failure were purely resolution, more memory would fix it. Our dense-plus-selector ablation tests this directly. We predict that a dense volume with argmin or soft-argmin still lands on the background, because the background is the global minimum at any resolution. Soft-argmin over two modes produces an intermediate depth that matches neither surface. If unconstrained dense argmin does recover glass, the claim narrows to memory efficiency alone.

Why not just opaquify inputs with GHOST and use any stereo network? GHOST's gains are real. However, it costs 0.38 s per view, which rules it out for multi-camera control at tens of hertz. Its per-view hallucination is not constrained to agree across views. We will measure that agreement directly, as the reprojection error of hallucinated texture across ClearPose view pairs. If it proves consistent and a faster distilled version is plausible, GHOST plus a standard volume becomes the method to beat, and our contribution shrinks to the two-layer output.

Decisive Checks

The project lives or dies on three measurements:

  1. Mode recall in the diagnostic study. A high rate means glass depth is usually among the top few minima, and the premise holds.
  2. Equal-budget comparison against the single-window pyramid on ClearPose's translucent-cover split. The multi-mode design must win at matched $B$.
  3. Rank stability of all methods across the three memory caps. If rankings reorder as the cap changes, published transparent-depth comparisons made at unmatched budgets are partly budget artifacts. That is the TB-Audit finding, reproduced in 3D perception.

A secondary, exploratory test feeds the two-layer depth into a Diffusion Policy on AutoBio's pipetting and tube-capping tasks, rendered from multiple views. For reference, AutoBio reports 18.2% to 42.7% and 1.7% to 8.3% success on these tasks for current models (AutoBio). This test asks whether better glass geometry moves closed-loop success at all.