MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation

Yong Yang, Qiong Chen, Yuan Feng, Tianlin HuangCVPR 2023 · DOI · arXiv

Introduction

Few-shot semantic segmentation (FSS) is a challenging computer vision task that requires a model to segment novel object classes in a "query" image based on only a handful of labeled "support" examples. The primary difficulty lies in overcoming intra-class variation—where the appearance of an object (e.g., a dog) varies significantly due to pose, lighting, or camera angle—and base-class bias, where models overfit to the specific categories seen during training, hindering generalization to novel classes.

The MIANet Architecture

To address these challenges, the Multi-Information Aggregation Network (MIANet) introduces a dual-pathway architecture that combines specific "instance" details with "general" category knowledge derived from word embeddings.

Comparison between existing few-shot segmentation methods and the proposed MIANet architecture.

General Information Module (GIM)

The GIM is a core innovation of MIANet. It utilizes pre-trained word embeddings (word2vec) to provide a canonical representation of a class. By merging the visual support prototype with a semantic word vector, the model creates a "General Class Prototype." This acts as a global anchor, supplementing the limited local information provided by the few-shot support set. To ensure this prototype is visually grounded, a triplet loss is employed to align language-based prototypes with visual features, pulling similar local features closer while pushing dissimilar background features away.

Hierarchical Prior Module (HPM)

To mitigate base-class bias, MIANet employs a non-parametric Hierarchical Prior Module. This module calculates pixel-level cosine similarity between support and query features across four different scales without using learned weights. By remaining class-agnostic, the HPM generalizes more effectively to unseen categories than traditional parametric approaches that might "memorize" training classes.

Information Fusion Module (IFM)

The final stage involves the IFM, which aggregates the multi-scale instance-level maps from the HPM and the general prototype from the GIM. This fusion allows the network to leverage both specific visual cues and broader semantic concepts to generate the final segmentation mask.

Experimental Evaluation

MIANet was evaluated on two major FSS benchmarks: PASCAL-5i and COCO-20i.

  • PASCAL-5i: MIANet achieved a 67.10% mIoU (1-shot) using a VGG16 backbone, representing a 2.69% improvement over previous state-of-the-art (SOTA) methods like BAM.
  • COCO-20i: On this more complex and diverse dataset, the model reached 45.69% mIoU (1-shot), outperforming BAM by 2.19%.

Qualitative Analysis

Visual results demonstrate MIANet's robustness against extreme perspective differences and partial occlusions. For instance, the model successfully segments objects like aeroplanes and birds even when the query image presents a vastly different viewpoint or scale than the support image.

Qualitative examples of MIANet's segmentation performance on novel classes like Bird and Aeroplane.

Conclusion

MIANet advances the state-of-the-art in few-shot semantic segmentation by effectively balancing instance-specific and general category information. By incorporating language as a stabilizer and utilizing non-parametric priors, the model reduces bias and improves generalization. Future work will focus on enhancing performance for small objects in complex scenes and further addressing extreme cases of pose and occlusion.

Summary updated

BibTeX
@inproceedings{yang2023mianet,
  author = {Yong Yang and Qiong Chen and Yuan Feng and Tianlin Huang},
  title = {{MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00689},
}
Figure