Stereo networks learn depth from pairs of images. They are trained mostly on rendered scenes, because rendering gives perfect depth labels for free. When these networks meet real cameras, their error jumps: on the real Booster dataset, error exceeds 40% of pixels and 80% on glass and shiny surfaces (Open Challenges in Deep Stereo: the Booster Dataset). The usual fix is to fine-tune on real, unlabeled footage using the network's own confident predictions as labels, known as pseudo-labels. That fix only works if we can tell which predictions are right. Current methods decide this with signals that fail under the same conditions that caused the gap. They use image-matching scores, which break on reflections and glare, or the network's own sense of certainty, which stays high exactly when the network has learned a simulator shortcut.
We propose to settle that question with geometry instead of appearance. During data collection only, a third calibrated camera is mounted next to the stereo pair. The synthetic-trained network predicts depth independently from two different camera pairs that share a reference image. A pixel's prediction is kept as a label only if both pairs place the same physical point at the same depth, which is the classical trinocular point-to-point check. The student network is then fine-tuned on those labels and deployed as an ordinary two-camera system. The two-sentence version: a third camera turns "does the network feel sure?" into "do two independent baselines agree on where this point is in 3D?", and that agreement is used to choose real-world training labels. The third camera is needed only while gathering adaptation data, never at test time.
Core Claim
Trinocular verification was built to reject false matches. A match between views 1 and 2 fixes a 3D point, and that point must project to one exact pixel in view 3 (Trinocular Stereovision: Recent Results; 3D Model Acquisition from Extended Image Sequences). We claim this check makes a better pseudo-label filter for synthetic-to-real adaptation than photometric or uncertainty-based filters, for one specific class of errors: errors that depend on viewpoint. These include repetitive textures, matching ambiguity along edges parallel to the baseline, and moving specular highlights on curved surfaces.
We also make a sharper prediction that limits the claim. The check cannot reject errors that are geometrically consistent across viewpoints. A planar mirror produces a virtual image that sits at one well-defined 3D location, so all three cameras agree on the wrong depth. Glass that shows the background behind it behaves the same way. The research file's suggestion that a third view "prunes non-Lambertian reflection artifacts" is therefore only half true. Part of the proposal is to measure which half holds, using Booster's material masks.
For a rectified rig with reference camera 1, pair baselines $b_{12}$ and $b_{13}$, and network disparities $d_{12}(p)$ and $d_{13}(p)$ at pixel $p$, the check reduces to the following test:
$$ \left| d_{13}(p) - \tfrac{b_{13}}{b_{12}}\, d_{12}(p) \right| < \tau , $$
Here $\tau$ is a tolerance in pixels. The form follows from depth being $f b / d$ for focal length $f$. For unrectified or temporally formed triplets, the same test is computed by transferring the point through the trifocal tensor. Both disparities come from the network's own predictions, so the test never compares pixel colors. That is the essential difference from photometric self-supervision.
Research Program
The pipeline has three parts. The first is a teacher pretrained on SceneFlow and run on two camera pairs sharing a reference view. The second is the agreement mask above, producing pseudo-labels: the accepted pixels, plus the baseline-weighted average of the two agreeing estimates. The third is a student initialized from the teacher. Following Robust Synthetic-to-Real Transfer for Stereo Matching, which showed that naive fine-tuning erodes general geometric knowledge, the student keeps a frozen copy of the teacher as a regularizer and an exponential-moving-average teacher that is re-labeled every few thousand steps. The backbone is a standard architecture from OpenStereo, so that every comparison changes only the label filter.
The geometry literature dictates two design choices that are the substantive part of the work.
The first is rig shape. If the three camera centers lie on a line, the two pairs share an epipolar direction. Horizontal repetitive texture then fools both pairs, and only the change in period between the two baselines distinguishes them. An L-shaped rig adds a vertical baseline, which breaks horizontal ambiguities and edge-aperture problems. Trinocular Geometry Revisited shows that collinear centers need two trilinear constraints rather than one to guarantee that rays truly meet. For general triplets, The Joint Image Handbook shows that trilinear constraints alone admit spurious matches near epipoles. We therefore implement the full pairwise-plus-trilinear test and compare collinear and non-collinear triplets directly. To run the vertical pair, the reference and third images are rotated so that a horizontally trained network can process them.
The second is where the third view comes from. A physical third camera is clean but costs hardware. A temporal third view, the next frame of the left camera with pose from odometry, is free but often near-collinear during forward driving. In that configuration Trinocular Geometry Revisited predicts the weaker "3+2" regime, and moving objects break the assumption that the scene is static. TVG-SLAM uses exactly such consecutive-frame trifocal constraints, though for camera tracking and with a 400 ms-per-frame dense matcher. We test whether temporal triplets can substitute for a rig, and where they fail.
The occlusion structure of triplets also matters. A point visible in pair (1,2) may be hidden in view 3. Rejected pixels are therefore not marked as wrong. They simply receive no label, and the student's loss on them comes only from the frozen-teacher regularizer.
Baselines
The naive reference is the SceneFlow-pretrained backbone evaluated with no adaptation. The Real-Time Self-Adaptive Deep Stereo paper reports "D1-all ~37%" on KITTI for its unadapted model, and Domain-Invariant Stereo Matching Networks reports 11.7% for synthetic-only GANet. Any adapted method must clearly improve on our own measured unadapted number for the same backbone.
The zero-shot competitor is DSMNet, which normalizes away color and contrast statistics so that a synthetic-only model transfers without adaptation. It reports a "6.5% error rate" zero-shot on KITTI. We run its normalization layers inside our backbone with no adaptation. This tests whether removing style differences is enough, which is what DSMNet's own premise claims.
The photometric self-adaptation competitor is MAD from Real-Time Self-Adaptive Deep Stereo. It warps one image into the other and minimizes the color difference, and reports error "under 3%" on KITTI after online adaptation. The classical proxy-teacher competitor is Unsupervised Adaptation for Deep Stereo. It labels real frames with Semi-Global Matching filtered by a learned confidence network, and reports KITTI 2015 Bad 3% falling from 7.46% to 4.21%.
The strongest unsupervised competitor is UCFNet from Digging Into Uncertainty-based Pseudo-label for Robust Stereo Matching. It filters pseudo-labels by pixel-level and region-level uncertainty from the matching-cost distribution. It reports KITTI 2015 error reduced "from 5.2% to 3.1%" with no target ground truth. It is the strongest competitor because it asks exactly our question, which pixels to trust, with an internal statistical signal instead of geometry.
Fairness is the crux. Every two-view baseline receives the same unlabeled frames, including the (1,3) pairs treated as additional ordinary stereo pairs. This separates the value of verification from the value of extra data. All methods also get the same backbone, step count, and hyperparameter search budget, tuned on a held-out real split. Beating UCFNet means lower D1-all on KITTI 2015 under identical backbone and data. It also means higher pseudo-label precision at equal coverage, measured against LiDAR or scanner ground truth.
DKT-Stereo serves as a supervised upper reference rather than a competitor, since it uses target ground truth. It reports ETH3D error of 2.60% after Booster fine-tuning, against 93.31% for naive fine-tuning. Our cross-domain retention should be read against that gap.
Two ablations isolate the key ingredient. In the first, the "second pair" is the same (1,2) pair under a different augmentation, so the filter becomes self-ensemble agreement with no new geometry. In the second, the geometric agreement test is replaced by a three-view photometric warping loss. If either ablation matches the full method, the third view's geometry is not what is doing the work.
Why not just…?
Why not just use left-right consistency on the existing pair? Checking that left-to-right and right-to-left disparities agree is free and widely used. It is the most dangerous alternative. Its weakness is that both directions use the same baseline and the same epipolar line, so a periodic texture or a horizontal edge produces the same false match in both directions. We include it as a filter at matched coverage. If its pseudo-label precision on ETH3D and KITTI equals the trinocular filter's precision, the idea fails and we will report that. We expect the gap to appear mainly on the L-shaped rig and on Booster's repetitive and specular-but-curved regions.
Why not just use temporal frames as the third view, with no extra hardware? This is a genuine option, and we test it as a variant rather than dismiss it. The objections are specific and measurable. Forward driving makes the triplet near-collinear, which falls into the weaker regime described in Trinocular Geometry Revisited. Moving objects violate static-scene agreement. Pose errors from odometry also enter the check. If temporal triplets reach comparable precision outside dynamic regions, the practical recommendation becomes "no third camera needed," and that would be a useful result in itself.
Why not just trust a better uncertainty estimate, or bleach style as DSMNet does? Both rest on the model's internal view of itself. DSMNet assumes the gap is mostly appearance. Booster's 80% error on glass and specular surfaces shows the gap is partly geometric, and normalization cannot change which 3D point two cameras appear to see. Uncertainty estimates such as UCFNet's are computed from the same cost volume that learned the shortcut. They can therefore be confidently wrong, and our precision-at-coverage experiment checks this directly. The honest risk on our side is shared shortcuts. Because both pairs share the reference image and the network, a domain-shift error that is consistent across viewpoints will pass our check as well. We measure how correlated the two pairs' errors are, using ground truth, before claiming independence.
Proposed Experiments
The first and cheapest decisive check needs no training. On real multi-camera data with ground truth, we run the SceneFlow teacher and compare four filters at matched coverage: the trinocular filter, left-right consistency, UCFNet-style uncertainty, and the self-ensemble ablation. We measure pseudo-label precision, meaning the fraction of accepted pixels within 1 and 3 pixels of ground truth. The data are triplets drawn from the synchronized four-camera rig in A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos, using both collinear and non-collinear camera choices, plus KITTI raw frames with LiDAR ground truth. On KITTI we use the grayscale and color cameras on the shared bar as a collinear triplet. Precision is broken down by Booster material class where a usable third view exists. Booster pairs a 12 Mpx primary camera with a 1.1 Mpx secondary, so this also probes unbalanced resolution. If the trinocular filter does not beat left-right consistency here, we stop.
The second check is adaptation itself. We adapt on unlabeled KITTI raw and evaluate D1-all on KITTI 2015, against the baselines and ablations above. We adapt separately on ETH3D-rig footage and evaluate there. Finally, we test retention by evaluating the KITTI-adapted student zero-shot on Middlebury 2014, ETH3D, and DrivingStereo, to confirm that adaptation does not cause the degradation documented by DKT-Stereo.
The third check tests the falsifiable prediction about reflections. On Booster's mirror and transparent masks, we predict that trinocular filtering gives no precision gain over left-right consistency, while on curved specular surfaces it gives a clear gain. A uniform gain across all materials would contradict our geometric account and would suggest the filter is acting as a generic ensemble. In that case the self-ensemble ablation should then close the gap.
References
- Open Challenges in Deep Stereo: the Booster Dataset
- Trinocular Stereovision: Recent Results
- 3D Model Acquisition from Extended Image Sequences
- Robust Synthetic-to-Real Transfer for Stereo Matching
- OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline
- Trinocular Geometry Revisited
- The Joint Image Handbook
- TVG-SLAM: Robust Gaussian Splatting SLAM With Tri-View Geometric Constraints
- Real-time self-adaptive deep stereo
- Domain-Invariant Stereo Matching Networks
- Unsupervised Adaptation for Deep Stereo
- Digging Into Uncertainty-Based Pseudo-Label for Robust Stereo Matching
- A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos