Stereo depth estimation recovers distance by finding the same physical point in a left and a right image. That search breaks in predictable ways. On a blank wall every candidate match looks equally good. On a mirror one wrong match looks clearly best, because the network finds the reflected scene. A painted perspective illusion fools single-image depth models while stereo handles it easily, and glare or water on the lens corrupts both. Current methods guard against these failures by blending stereo matching with a single-image ("monocular") depth foundation model inside a learned refinement loop. That loop has to infer, implicitly and pixel by pixel, which cue to trust. We propose to make that decision explicit. A network first predicts why each region is hard, choosing among textureless, specular, transparent, monocular trap, lens hazard, occluded, or ordinary. A separate fusion stage then combines the two cues according to that diagnosis. The diagnosis can be trained without any real labels, because a procedural synthetic renderer knows the material and hazard behind every pixel for free.
The second ingredient comes from multimodal reasoning research. Vision-language models show a "seesaw": training that improves perception damages reasoning and vice versa, because one final-answer loss cannot tell which stage failed (Bad Seeing or Bad Thinking?). We expect the same confusion inside end-to-end stereo fusion. We therefore separate how the "seeing" stage (diagnosis) and the "deciding" stage (fusion) receive training signal. Unlike that work, we route the diagnosis through a dense map of categories rather than through text, since text is the bottleneck that paper itself concedes cannot carry continuous geometry.
Core Claim
The key fact is that the two standard warning signs disagree with the truth in opposite directions. A flat matching cost means stereo is uninformative and the monocular prior should take over. A mirror often produces a sharp but wrong minimum. A monocular trap produces a confident but wrong monocular prediction. Any gate built on confidence or cost flatness alone will therefore trust the wrong cue in exactly the regions where field deployment fails.
Our claim is that a per-pixel cause map, learned only from procedurally generated synthetic data, transfers zero-shot to real images well enough to beat implicit fusion on specular and transparent surfaces without losing ground on monocular traps. We further claim that training the diagnosis separately from the disparity loss avoids a seesaw between hazardous and ordinary regions.
Relation to Prior Work
Three works motivate the proposal by documenting the complementary failures. Open Challenges in Deep Stereo: the Booster Dataset builds a 12-megapixel indoor benchmark of mirrors and glass and reports bad-2 error above 40% overall and above 80% on transparent and specular pixels for leading networks. Stereo Anywhere adds Depth Anything v2 normals as a second correlation volume, truncates stereo matching at reflective surfaces, and introduces MonoTrap, where the monocular model reaches 53.46% AbsRel against Stereo Anywhere's 3.50%. DEFOM-Stereo injects foundation features and a recurrent scale map, and reports regional scale inconsistencies inside single scenes. Both fusion models are our strongest competitors. Both decide whom to trust inside a recurrent update, with no signal saying what the region is. That is the step we replace with an explicitly supervised diagnosis.
Synthetic-data work supplies the method for generating the diagnosis labels. What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation? shows that crude but diverse procedural data such as Flying Chairs can beat more realistic renders, and that simulating camera artifacts like Bayer interpolation improves KITTI by 5%. Diversity is Definitely Needed reaches the same diversity-over-realism conclusion for classification. Following both, we procedurally randomize mirrors, glass, featureless walls, trompe-l'oeil planes, glare, droplets and fog rather than pursuing photorealism.
Some prior work constrains the design. Improving Deep Stereo Network Generalization with Geometric Priors cuts Middlebury MAE from 5.37 to 3.21 with image-guided smoothness. Smoothness alone, however, propagates the wrong depth across mirrors, so we keep that prior only for the textureless class. Robust Synthetic-to-Real Transfer for Stereo Matching shows that fine-tuning on real Booster labels sends ETH3D error to 93.31%, versus 2.60% with its teacher-student filtering. That result argues for a purely synthetic training route and also gives us a real-data reference point. GeoSense finds that feeding geometry to a multimodal model unconditionally harms it, and that the model should request geometry only when needed. Our fusion applies the same "on demand" logic to the monocular prior. WildDash and DrivingStereo show that scene-wide averages hide localized failures, such as tunnels at 19.6 mIoU and stereo error rising sharply beyond 40 m. We therefore report every metric stratified by cause and by distance.
Research Program
We will build a synthetic hazard set on top of SceneFlow. Each rendered stereo pair carries ground-truth disparity and a per-pixel cause label. The cause classes are textured-Lambertian, textureless, specular, transparent, monocular trap (planar surfaces painted with perspective-consistent content), lens hazard (glare, droplets, dirt, applied after rendering together with Bayer artifacts), and occluded. For mirrors and glass, the ground-truth disparity is the surface itself, matching the Booster convention.
The network keeps the two-branch design of Stereo Anywhere: a RAFT-Stereo correlation branch, and a scaled Depth Anything v2 branch. It adds a light diagnosis head that reads image features and statistics of the stereo correlation volume and outputs class probabilities $p_c(x)$ for each pixel $x$ and cause $c$. Each class has its own small fusion expert $f_c$. For example, the specular expert truncates the cost volume at the monocular surface, the textureless expert relies on monocular shape plus stereo-derived scale, and the trap expert relies on stereo alone. The final disparity is
$$D(x)=\sum_c p_c(x)\, f_c\big(S(x), M(x)\big),$$
where $S$ is the stereo estimate and $M$ is the scaled monocular estimate.
The credit-assignment rule is simple. The diagnosis head is trained only with the cause cross-entropy, and a stop-gradient blocks the disparity loss from reaching it. The experts and the refinement stage are trained with the disparity loss. A disparity error therefore can never "teach" the diagnosis that a mirror is a wall, which is the dense-geometry analogue of protecting verified perception tokens in MoCA. We will also train an end-to-end variant, in which the disparity loss reaches the diagnosis head, to test whether the separation actually matters.
Baselines
The naive reference is RAFT-Stereo trained on SceneFlow alone and evaluated zero-shot. It anchors how large the gap is without any prior.
The strongest existing method is Stereo Anywhere, which reports 6.96% bad-2 on Middlebury 2014, an average error of 1.21 on Booster (down from 3.59), and 3.50% AbsRel on MonoTrap. We take these numbers from Stereo Anywhere. We will reproduce it from its published configuration, and we will also retrain it on exactly our SceneFlow-plus-hazard data with the same iteration count, resolution and tuning budget. Only the retrained version is a fair comparison, because otherwise any gain could come from our data rather than from diagnosis. Beating it means lower bad-2 on Booster's specular and transparent pixels by a margin larger than the spread across three seeds, while staying within the seed spread of 3.50% on MonoTrap and 6.96% on Middlebury. DEFOM-Stereo, which the file reports cuts Middlebury 2-pixel error by about one-third, is retrained under the same protocol as a second fusion competitor.
Two ablations isolate the key ingredient. A flatness gate replaces the learned diagnosis with a hand-designed rule: trust the monocular cue in proportion to the entropy of the correlation curve. A no-label variant keeps our architecture and data but drops the cause supervision, so the mixture is learned implicitly. An oracle run feeds ground-truth cause maps on synthetic validation data and on Booster's material annotations, giving an upper bound for a perfect diagnosis. DKT-Stereo, fine-tuned on Booster, serves as the real-label reference. It reports 2.60% on ETH3D after Booster fine-tuning. We ask whether our label-free model matches its cross-domain error.
Why not just…?
Why not just add the synthetic hazards to Stereo Anywhere's training and skip the diagnosis? This is the most serious threat. Domain randomization alone may teach the implicit refinement loop everything a cause map provides. The retrained Stereo Anywhere and the no-label ablation test this directly. If either matches our model on Booster's specular and transparent pixels, the claim fails, and the contribution reduces to the hazard dataset.
Why not just gate on matching-cost flatness or predicted confidence? Confidence signals mislead in a known direction. A reflection produces a confident wrong match, and a perspective illusion produces a confident wrong monocular depth. We will measure the peak sharpness of the correlation curve on Booster mirror pixels. If mirror costs turn out to be as flat as those on textureless walls, the flatness gate would suffice and we would say so. The flatness-gate baseline settles this empirically.
Why not just ask a vision-language model to find the mirrors and glass, then mask them? That approach routes geometry through a text or region bottleneck, which Bad Seeing or Bad Thinking? concedes is lossy. Multi-turn region prompting also fails whenever the first grounding step is wrong (DCoT). Still, we will run a variant that takes transparency and specularity masks from a prompted multimodal model and feeds them to the same experts. If it matches our learned diagnosis, the synthetic-label route loses its justification.
Why not just fine-tune on real field data? Real fine-tuning destroys cross-domain generalization: ETH3D error reaches 93.31% after Booster fine-tuning (Robust Synthetic-to-Real Transfer for Stereo Matching). Real labels for glass also required painting the scenes (Booster). DKT-Stereo remains a reference point rather than a competitor in the same setting.
Proposed Experiments
Every model trains on synthetic data only and is evaluated zero-shot on Middlebury 2014, ETH3D, KITTI 2012/2015, Booster, MonoTrap, and the DrivingStereo weather splits. We report bad-2 and average error per cause class using Booster's material masks, plus a small set of real cause masks we will annotate on MonoTrap and DrivingStereo fog and rain frames. On DrivingStereo we additionally report error by distance band. The file has no absolute fog or rain numbers for DKT-Stereo on DrivingStereo, so we will run it ourselves.
There are four decisive checks. The first is whether diagnosis beats the retrained Stereo Anywhere on specular and transparent pixels without losing on MonoTrap. The second is whether the cause map transfers: its real-image accuracy, compared with the oracle run, tells us whether the gains come from correct diagnosis or from extra capacity. The third is the seesaw test. Comparing the stop-gradient and end-to-end variants, we ask whether improvements on hazard pixels come at the cost of error on ordinary textured pixels. The fourth is whether the flatness gate already captures the benefit, which would reduce the proposal to a data contribution.
References
- Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
- Open Challenges in Deep Stereo: the Booster Dataset
- Stereo Anywhere: Robust Zero-Shot Deep Stereo Matching Even Where Either Stereo or Mono Fail
- DEFOM-Stereo: Depth Foundation Model Based Stereo Matching
- What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
- Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
- Improving Deep Stereo Network Generalization with Geometric Priors
- Robust Synthetic-to-Real Transfer for Stereo Matching
- GeoSense: Internalizing Geometric Necessity Perception for Multimodal Reasoning
- WildDash - Creating Hazard-Aware Benchmarks
- DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models