Stereo Features as the Missing Local Signal for Real-Time Open-Vocabulary Segmentation

Vision-language models such as CLIP learn to match a whole image to a caption. They are good at saying what is in a scene and poor at saying exactly where: their per-patch outputs are coarse, and the boundaries of objects smear across neighbors. Fast stereo networks have the opposite profile. These networks estimate depth by measuring how far each pixel shifts between a left and a right camera image, a shift called disparity. They find precise geometric edges at high resolution but have no idea what anything is, so they also cannot tell a painted lane line from a real depth edge. This note proposes one compact model that joins the two. A real-time stereo network does double duty: it estimates disparity, and it also serves as the high-resolution "side network" that tells a frozen, low-resolution CLIP where object boundaries are. The coarse open-vocabulary masks that come back are then used to guide the stereo network's own edge-refinement step in place of raw color edges.

The opportunity is practical. Robots, drones and headsets with stereo cameras already pay for a disparity network on every frame. If that same computation can sharpen language-queryable masks, then open-vocabulary 3D perception costs little more than stereo alone. The current options are a second heavy segmenter, or joint models that run at about 3 frames per second with a fixed 19-class vocabulary. The scientific question is narrower and testable: do binocular matching features carry boundary information that monocular features and simple depth-guided filtering do not, and does that information close CLIP's localization gap better than the alternatives at equal latency?

Core Claim

In a stereo pair, the high-resolution local signal that CLIP lacks is already being computed by the disparity network, in the form of matching costs and disparity discontinuities. We claim three things. First, a lightweight stereo backbone in the style of StereoNet or BANet can replace the separate high-resolution side network of SAN. Second, it produces class-agnostic mask proposals whose boundaries follow true depth edges, while the CLIP encoder stays frozen and runs at low resolution. Third, the resulting masks give the stereo refinement stage a guidance signal that ignores flat surface texture, which reduces false depth steps on road markings. The new element is the shared geometric side network and the closed loop between the two outputs. Neither adding semantics to stereo nor adding depth to segmentation is new on its own.

Relation to Prior Work

Two papers motivate the proposal by showing that the coupling helps but is too expensive. TiCoSS trains segmentation and stereo jointly with gated fusion between the two decoders. It reports a 10.57% mIoU gain over S3M-Net and a 3.64% reduction in disparity end-point error on KITTI 2015, which is evidence that each task helps the other. It uses 385M parameters, runs at about 3 FPS, and covers only closed-set classes. We keep its finding but use a frozen language model for the vocabulary and a real-time stereo network for the geometry. StereoVLA feeds a FoundationStereo cost volume, together with SigLIP and DINOv2 tokens, into a language model for robot control. It shows that stereo geometry matters where monocular inputs fail: 30% success on 1–2 cm objects against 0% for the baselines. The information flows one way, at 224×224 and offline rates. Our flow runs in both directions and is designed for edge latency.

Two papers supply the method. SAN freezes CLIP and attaches an 8.4M-parameter side vision transformer. The side network proposes masks at high resolution and injects attention biases so that CLIP pools its features inside each mask. We reuse this attention-bias interface exactly, but the side network is the stereo backbone fed by the left features and the cost volume, rather than an RGB-only transformer. StereoNet builds a cost volume at 1/8 resolution and recovers edges with a refinement step guided by the color image, a learned version of edge-aware upsampling. We keep that structure and change only what guides the refinement.

The remaining papers constrain the design. Vireo conditions open-vocabulary segmentation on monocular DepthAnything-V2 features, reaching 32.72% mIoU on Cityscapes→DELIVER. It shows that depth helps segmentation and raises the obvious question of whether monocular depth would do just as well, which we test directly. OV-NeRF cleans CLIP maps with SAM regions and multi-view consistency, but it needs per-scene optimization. DenseCLIP shows that CLIP feature maps keep usable spatial information. It also fine-tunes the backbone, which is a regime that the open-vocabulary literature, including the TCSVT 2026 fine-grained alignment work, links to a loss of zero-shot ability. That evidence is the reason we keep CLIP frozen.

Research Program

The model has three parts. The first is a stereo backbone in the style of BANet-2D: a shared feature extractor, a correlation volume at 1/4 or 1/8 resolution, and 2D aggregation, which yields a coarse disparity map and an aggregated cost tensor. The second is a mask head attached to this backbone. It reads the left-image features, the aggregated cost, and a disparity-gradient map, and it emits a small number of class-agnostic mask proposals plus SAN-style attention biases. The third is a frozen CLIP ViT-B run on a downsampled left image. The attention biases make CLIP pool its features over each proposal, and each proposal is classified against arbitrary text prompts.

The loop closes in the refinement stage. The final edge-aware upsampling of disparity receives two guidance signals: the color image, as in StereoNet, and the soft mask assignments. The refinement can therefore learn to place depth edges where proposals change, and to ignore color edges that fall inside a single proposal such as a road.

Training uses disparity supervision together with mask and classification losses on a set of seen classes only. Held-out classes test whether open-vocabulary transfer survives. The open-vocabulary classifier never sees held-out names during training. CLIP's weights are never updated. Latency is measured end to end on a Jetson Orin and on a Snapdragon 8 Gen 3, so the numbers can be compared with BANet's reported 45 ms on the latter.

The analysis has two parts. The first asks where the boundary gains come from. We measure mask boundary quality separately at pixels that are depth discontinuities and at pixels that are color-only edges. The second asks whether the refinement improvement is concentrated on coplanar textured surfaces. These are pixels on road or sidewalk with strong color gradients, identified from semantic labels.

Baselines

The naive reference point runs BANet-2D for disparity and SAN independently on the left image, then projects SAN's masks onto the depth map. This is the cascade any practitioner would build. Its latency is the sum of two networks, and the stereo branch gets no semantic help. It is run with both components trained on the same data and seen-class split as our model. If our model matches its segmentation quality at clearly lower total latency, or beats it at equal latency, that is a positive result.

The strongest open-vocabulary competitor is SAN with its original RGB side network, trained on the same seen classes and given the same tuning budget. SAN reports 12.4 mIoU on ADE20K-847 when trained on COCO-Stuff (SAN). ADE20K has no stereo pairs, however, so that number is not directly comparable. There is no reported result on Lacuna for SAN on Cityscapes or KITTI stereo splits, and we will rerun it. Beating it means higher unseen-class mIoU and boundary F-score at no greater latency.

The strongest joint semantic-stereo model is TiCoSS. The file reports its KITTI 2015 results only as relative gains (10.57% mIoU over S3M-Net, 3.64% EPE reduction) and gives no absolute values. We will retrain it on our seen classes and evaluate it as a closed-set upper reference for seen-class mIoU and disparity. We do not expect to beat it on seen classes. The bar is to stay within a few points while running more than ten times faster and also handling unseen classes.

For disparity, the reference point is StereoNet, which reports an end-point error of 1.10 on Scene Flow (StereoNet), and BANet-2D on KITTI 2015 under our own retraining. Beating them means lower end-point error and D1-all on coplanar textured regions with no regression elsewhere.

Two ablations isolate the key ingredient. In the first, the mask head reads only the left-image features of the stereo backbone, with no cost volume and no disparity gradient. This tests whether binocular geometry matters or whether any shared convolutional backbone would do. In the second, the refinement uses color guidance only. This tests the return path from masks to disparity.

Why not just…?

Why not just upsample CLIP's coarse map with a depth-guided bilateral filter? This is the most serious threat. Once disparity exists, a training-free joint bilateral upsampling of DenseCLIP-style score maps, guided by disparity rather than color, already snaps masks to depth edges at almost no cost. We include it as a baseline, applied to the same frozen CLIP outputs and our own disparity. If it matches our learned head on unseen-class boundary F-score, then the learned side network is not worth its cost, and the contribution shrinks to that filter plus the refinement loop. We expect it to fail where filtering cannot help: where CLIP's coarse patch assigns the wrong class to an entire small object, and where two adjacent objects have the same depth. Attention-bias pooling can re-classify a region in those cases, but filtering can only move its edges. The per-object and boundary analysis will show which case holds.

Why not just add monocular depth from DepthAnything-V2, as Vireo does? Monocular depth would remove the need for a second camera. But it adds a large transformer to the latency budget, and its depth edges are inferred from appearance, so they can inherit the same color-texture confusions we are trying to remove. Evidence from StereoVLA shows that monocular inputs fail at fine geometry. That evidence concerns manipulation, not segmentation, so we test this alternative directly. The SAN side network is fed RGB plus DepthAnything-V2 depth and compared with our model at matched latency and at matched accuracy.

Why not just use SAM proposals pooled over CLIP features? SAM gives sharp class-agnostic masks, and OV-NeRF shows that they clean CLIP relevancy maps. SAM's heavy encoder does not fit an edge frame budget, and its boundaries follow appearance, not geometry. We report SAM-with-CLIP as an offline accuracy ceiling for boundaries, not as a real-time competitor.

Why not just use a closed-set joint model like TiCoSS? If the deployment vocabulary is fixed, a closed-set joint model is a fine answer, and it can be made smaller. The open-vocabulary setting matters for robots and headsets that are queried with arbitrary object names. The held-out-class split measures exactly this, and closed-set models score zero on it by construction.

Proposed Experiments

Disparity pretraining uses Scene Flow (Mayer et al.). Joint training uses Cityscapes stereo pairs with semantic labels and vKITTI2, both of which TiCoSS also uses. Real-world evaluation uses KITTI 2015 (Geiger et al.) disparity and semantic annotations. We hold out a set of classes from the training labels, for example rider, bus, traffic sign, pole and terrain, so that the split contains both thin structures and textureless regions. Segmentation is evaluated on seen and unseen mIoU and on boundary F-score. Boundary F-score is split into pixels at depth discontinuities and pixels at color-only edges. Disparity is evaluated on end-point error and D1-all, overall and on coplanar textured road pixels. Latency is measured on Jetson Orin and Snapdragon 8 Gen 3.

The decisive checks are three. First, the cost-volume ablation must lose unseen-class boundary F-score at depth discontinuities. If it does not, the geometry claim is false. Second, the learned head must beat depth-guided bilateral upsampling of CLIP maps on unseen-class mIoU, not only on boundaries. If it does not, the simpler method wins and we report that outcome. Third, mask-guided refinement must lower D1-all on coplanar textured regions relative to color-only guidance, without raising it at true object edges. The whole system should also stay within a modest margin of BANet-2D's own latency on the same chip. If the CLIP pass dominates latency, we will test running CLIP every few frames and propagating its labels through the stereo features, and report how accuracy decays with the update interval.