Introduction
One-shot semantic segmentation aims to segment objects of a previously unseen category given only a single labeled support image. Traditional methods typically compress the support image into a global feature vector to guide the segmentation of the query image. However, this "one-to-many" matching approach often loses critical spatial information and fails to capture the complex relationships between different object parts.
To address these limitations, we present Pyramid Graph Networks (PGNet). PGNet shifts the paradigm from global descriptors to a "many-to-many" graph matching strategy. By establishing element-to-element correspondences between query and support images across multiple scales, the model preserves structural information and achieves more precise segmentation.
Methodology: The PGNet Architecture
The core of PGNet is the establishment of a bipartite graph between the query image and the foreground regions of the support image.
Graph Attention Unit (GAU)
The GAU is the fundamental component that performs graph reasoning. It treats pixels or regions as nodes and computes attention weights (correlation scores) between every query node and every support node. This allows the model to identify which specific parts of the support image (e.g., a bird's wing) correspond to regions in the query image.
Pyramid Reasoning Structure
Objects in natural scenes vary significantly in scale. PGNet employs a pyramid of GAUs to handle this diversity:
- Fine-grained Branch: Operates on high-resolution feature maps for pixel-to-pixel matching.
- Coarse-grained Branches: Use adaptive pooling to create region-based nodes, enabling the model to reason about larger object parts and global context.
- Global Branch: A specialized branch with "equal weights" provides a baseline global guidance similar to traditional methods.
The outputs from these multi-scale branches are aggregated and refined through residual convolutions and an Atrous Spatial Pyramid Pooling (ASPP) module to generate the final segmentation mask.
Experiments and Results
The proposed model was evaluated on the PASCAL VOC 2012 dataset, supplemented by the Semantic Boundaries Dataset (SBD). The dataset was split into four folds to test generalization to unseen classes.
Performance Comparison
PGNet demonstrates superior performance compared to previous state-of-the-art methods. On the PASCAL-5i benchmark, PGNet achieved a mean Intersection over Union (mIoU) of 56.0% in the 1-shot setting and 58.5% in the 5-shot setting. This improvement highlights the effectiveness of graph-based many-to-many matching over global vector guidance.
Interpretability and Qualitative Analysis
A significant advantage of the graph attention mechanism is its interpretability. By visualizing the attention maps, we can observe exactly which parts of the support image the model prioritizes when segmenting the query image.
As shown in the qualitative results, the model successfully identifies and segments objects even in complex scenes with varying poses and scales. The attention maps confirm that the model learns meaningful correspondences between the support and query sets.
Conclusion
Pyramid Graph Networks represent a significant step forward in one-shot semantic segmentation. By replacing lossy global descriptors with a multi-scale graph reasoning framework, PGNet effectively captures the structural relationships necessary for identifying novel object categories. Future work may involve exploring the use of background context in the support set and automating the selection of pyramid scales.
Summary updated