Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation

Hongli Liu, Yu Wang, Shengjie ZhaoCVPR 2026 Findings 2026 · OpenReview

TL;DR

Few-shot segmentation often fails when the support and query images display the same object from drastically different viewpoints. This paper introduces VINE (View-Informed NEtwork), a framework that models structural consistency across viewpoints using a dual spatial-view graph alongside a query-adaptive foreground modulation mechanism. By combining structure-sensitive features with the Segment Anything Model (SAM), VINE achieves state-of-the-art segmentation accuracy with high parameter efficiency.

What Problem Does This Paper Address?

Few-Shot Segmentation (FSS) task templates require a model to segment a novel object category in a query image given only a handful of annotated support examples. A persistent bottleneck in FSS is cross-view inconsistency and structural misalignment. If the support image shows a car from the front, but the query image shows it from the side, standard prototype-matching methods often break down.

Even when leveraging foundation models like SAM, instance-centric saliency maps can propagate poorly across mismatched poses, leading to prototype drift and boundary ambiguity. The authors set out to answer a fundamental question: How can we explicitly couple cross-view structural alignment with robust, query-adaptive foreground discrimination?

View-induced challenges and modeling strategies

Methods

VINE adopts a dual-encoder architecture that extracts semantic-rich representations from a frozen SAM encoder and structure-sensitive features from a ResNet backbone. It processes these features through two primary components:

  1. Spatial-View Graph Alignment (SVGA): To simulate viewpoint diversity, VINE generates perturbed support views using homography-based transformations. It then constructs a dual-graph architecture on the ResNet features:
  2. Spatial Graph: Captures local geometric topology within each view.
  3. View Graph: Connects features across different perspectives using a star topology (anchoring auxiliary views to the original view) to propagate view-invariant structural semantics. A prototype consistency loss ($\mathcal{L}_{proto}$) enforces alignment between query features and these view-modeled prototypes.
  4. Discriminative Foreground Modulation (DFM): To resolve foreground ambiguity, DFM measures the feature discrepancy between the support and query images. This contrast yields a query-adaptive discriminative prior that highlights target object regions while suppressing background distractors.

Finally, learnable prompt tokens aggregate these refined features through masked cross-attention, producing a Visual Reference Prompt (VRP) that guides the SAM decoder to output the final segmentation mask.

Figure 2. An overview of our proposed framework. We incorporate Spatial-View Graph Alignment (SVGA) to capture structural consis- tency across views, and Discriminative Foreground Modulation (DFM) to highlight foreground-relevant features, collaboratively improving the quality of support-query prototypes and guiding accurate mask prediction.

Datasets

The authors evaluated VINE on two widely recognized FSS benchmarks:

  • PASCAL-$5^i$: Comprising 20 object categories from PASCAL VOC 2012 and SDS, divided into 4 disjoint folds for cross-validation.
  • COCO-$20^i$: A more challenging dataset featuring 80 categories from MS COCO, also evaluated across 4 disjoint folds.

Key Insights and Ideas

  • Decoupling Semantics and Geometry: Relying solely on SAM's appearance-based prompts is insufficient for FSS under extreme pose changes. By running a parallel, structure-focused ResNet branch optimized via graph neural networks, VINE successfully decouples semantic matching from geometric alignment.
  • The Power of Star Topologies: Instead of using a fully connected view graph—which often suffers from over-smoothing—VINE uses a star topology where the original view acts as the central hub. This stabilizes message passing and preserves a consistent alignment space.
  • Cross-Class Structural Stability: In extreme "cross-class" testing scenarios (e.g., training on dogs but testing on motorbikes), conventional models completely collapse. VINE, however, retains significant localization capability because its SVGA module maps fundamental geometric continuity rather than just memorizing class-specific textures.

Key Results

VINE delivers consistent performance gains across multiple backbones and shot configurations:

  • PASCAL-$5^i$ Performance: VINE achieves 74.2% mIoU in the 1-shot setting and 75.1% mIoU in the 5-shot setting (using a ResNet-50 backbone), outperforming the strong FCP baseline by +2.1 and +1.1 points, respectively.
  • COCO-$20^i$ Performance: On this highly cluttered dataset, VINE reaches 53.7% mIoU (1-shot) and 59.3% mIoU (5-shot), demonstrating robust generalization.
  • Parameter Efficiency: VINE achieves these state-of-the-art results with only 27.6M total parameters, of which merely 3.9M are learnable. This proves that its performance stems from architectural innovation rather than brute-force scaling.

Parameter efficiency comparison Pseudo-mask quality under different methods

Limitations and Future Work

Not discussed in the paper.

Summary updated

Abstract

Few-shot segmentation (FSS) has gained significant attention for its ability to generalize to novel classes with limited supervision, yet remains challenged by structural misalignment and cross-view inconsistency under large appearance or viewpoint variations. This paper tackles these challenges by introducing VINE (View-Informed NEtwork), a unified framework that jointly models structural consistency and foreground discrimination to refine class-specific prototypes. Specifically, VINE introduces a spatial-view graph on backbone features, where the spatial graph captures local geometric topology and the view graph connects features from different perspectives to propagate view-invariant structural semantics. To further alleviate foreground ambiguity, we derive a discriminative prior from the support-query feature discrepancy to capture category-specific contrast, which reweights SAM features by emphasizing salient regions and recalibrates backbone activations for improved structural focus. The foreground-enhanced SAM features and structurally enriched ResNet features are progressively integrated through masked cross-attention, yielding class-consistent prototypes used as adaptive prompts for the SAM decoder to generate accurate masks. Extensive experiments on multiple FSS benchmarks validate the effectiveness and robustness of VINE, particularly under challenging scenarios with viewpoint shifts and complex structures.
BibTeX
@inproceedings{liu2026unify,
  author = {Hongli Liu and Yu Wang and Shengjie Zhao},
  title = {{Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=zgyGFXNTaA},
}
Figure