TL;DR
Few-shot segmentation often fails when the support and query images display the same object from drastically different viewpoints. This paper introduces VINE (View-Informed NEtwork), a framework that models structural consistency across viewpoints using a dual spatial-view graph alongside a query-adaptive foreground modulation mechanism. By combining structure-sensitive features with the Segment Anything Model (SAM), VINE achieves state-of-the-art segmentation accuracy with high parameter efficiency.
What Problem Does This Paper Address?
Few-Shot Segmentation (FSS) task templates require a model to segment a novel object category in a query image given only a handful of annotated support examples. A persistent bottleneck in FSS is cross-view inconsistency and structural misalignment. If the support image shows a car from the front, but the query image shows it from the side, standard prototype-matching methods often break down.
Even when leveraging foundation models like SAM, instance-centric saliency maps can propagate poorly across mismatched poses, leading to prototype drift and boundary ambiguity. The authors set out to answer a fundamental question: How can we explicitly couple cross-view structural alignment with robust, query-adaptive foreground discrimination?
Methods
VINE adopts a dual-encoder architecture that extracts semantic-rich representations from a frozen SAM encoder and structure-sensitive features from a ResNet backbone. It processes these features through two primary components:
- Spatial-View Graph Alignment (SVGA): To simulate viewpoint diversity, VINE generates perturbed support views using homography-based transformations. It then constructs a dual-graph architecture on the ResNet features:
- Spatial Graph: Captures local geometric topology within each view.
- View Graph: Connects features across different perspectives using a star topology (anchoring auxiliary views to the original view) to propagate view-invariant structural semantics. A prototype consistency loss ($\mathcal{L}_{proto}$) enforces alignment between query features and these view-modeled prototypes.
- Discriminative Foreground Modulation (DFM): To resolve foreground ambiguity, DFM measures the feature discrepancy between the support and query images. This contrast yields a query-adaptive discriminative prior that highlights target object regions while suppressing background distractors.
Finally, learnable prompt tokens aggregate these refined features through masked cross-attention, producing a Visual Reference Prompt (VRP) that guides the SAM decoder to output the final segmentation mask.
Datasets
The authors evaluated VINE on two widely recognized FSS benchmarks:
- PASCAL-$5^i$: Comprising 20 object categories from PASCAL VOC 2012 and SDS, divided into 4 disjoint folds for cross-validation.
- COCO-$20^i$: A more challenging dataset featuring 80 categories from MS COCO, also evaluated across 4 disjoint folds.
Key Insights and Ideas
- Decoupling Semantics and Geometry: Relying solely on SAM's appearance-based prompts is insufficient for FSS under extreme pose changes. By running a parallel, structure-focused ResNet branch optimized via graph neural networks, VINE successfully decouples semantic matching from geometric alignment.
- The Power of Star Topologies: Instead of using a fully connected view graph—which often suffers from over-smoothing—VINE uses a star topology where the original view acts as the central hub. This stabilizes message passing and preserves a consistent alignment space.
- Cross-Class Structural Stability: In extreme "cross-class" testing scenarios (e.g., training on dogs but testing on motorbikes), conventional models completely collapse. VINE, however, retains significant localization capability because its SVGA module maps fundamental geometric continuity rather than just memorizing class-specific textures.
Key Results
VINE delivers consistent performance gains across multiple backbones and shot configurations:
- PASCAL-$5^i$ Performance: VINE achieves 74.2% mIoU in the 1-shot setting and 75.1% mIoU in the 5-shot setting (using a ResNet-50 backbone), outperforming the strong FCP baseline by +2.1 and +1.1 points, respectively.
- COCO-$20^i$ Performance: On this highly cluttered dataset, VINE reaches 53.7% mIoU (1-shot) and 59.3% mIoU (5-shot), demonstrating robust generalization.
- Parameter Efficiency: VINE achieves these state-of-the-art results with only 27.6M total parameters, of which merely 3.9M are learnable. This proves that its performance stems from architectural innovation rather than brute-force scaling.

Limitations and Future Work
Not discussed in the paper.
Summary updated