Fine-Grained Prototypes Distillation for Few-Shot Object Detection

Zichen Wang, Bo Yang, Haonan Yue, Zhenghao MaAAAI 2024 · DOI

Abstract

Few-Shot Object Detection (FSOD) remains a significant challenge in computer vision, requiring models to identify novel categories from extremely limited labeled examples. Traditional meta-learning approaches typically condense support information into a single, global class-level prototype, which often lacks the local context necessary for precise detection. This paper introduces Fine-Grained Prototypes Distillation (FPD), a novel framework that captures distinctive local features through a transformer-based distillation process. By employing fine-grained prototypes, a balanced sampling strategy, and non-linear feature fusion, FPD achieves new state-of-the-art results on standard benchmarks, including PASCAL VOC and MS COCO.

The Challenge of Global Prototypes

The primary hurdle in FSOD is the "cold start" problem: training a model to detect rare categories with only a handful of images. Most existing meta-learning methods utilize a support branch to generate a single class-level prototype—a mathematical average of features from the available examples.

Comparison between conventional FSOD methods using class-level prototypes and the proposed FPD using fine-grained prototypes.

However, a single vector is often too "blurry" to represent complex objects. It fails to capture intricate details and local context, making the model sensitive to variations in background, pose, or lighting. The authors argue that capturing "distinctive local context" is essential for robust detection in data-scarce environments.

Proposed Methodology: Fine-Grained Prototypes Distillation

To address the limitations of global averages, the FPD framework introduces three core components designed to enhance feature representation and fusion.

The overall architecture of the proposed FPD framework, highlighting the Fine-Grained Feature Aggregation (FFA) and Non-Linear Fusion (NLF) modules.

1. Fine-Grained Feature Aggregation (FFA)

Instead of a single prototype, the FFA module utilizes learnable "feature queries" (inspired by the DETR architecture) to distill representative local features from support images. These fine-grained prototypes are assigned to the query image via cross-attention. This allows the model to perform part-based matching—for instance, identifying a bird in a query image by specifically matching its beak or wings to the distilled prototypes of the support set.

2. Balanced Class-Agnostic Sampling (B-CAS)

Feature fusion in FSOD often suffers from class bias. Previous methods either focused solely on the target class or sampled other classes randomly, which can lead to instability. B-CAS ensures a balanced mixture of positive and negative support prototypes during training. This prevents the model from being overwhelmed by irrelevant "negative" data while forcing it to learn the discriminative boundaries between different classes.

3. Non-Linear Fusion (NLF)

While standard models often rely on simple linear operations (like multiplication) for feature fusion, NLF employs a more sophisticated approach. It combines element-wise multiplication, subtraction, and concatenation, processing them through independent layers. This allows the model to capture more complex, non-linear relationships between the query and support features.

Knowledge Transfer Strategy

To facilitate the transition from base classes (with abundant data) to novel classes (with few shots), the authors implement a clever initialization strategy. They duplicate the most "compatible" feature queries from the base training phase to initialize the queries for novel classes, ensuring the model starts with a strong feature-extraction foundation.

Experimental Results

The FPD method was evaluated on the PASCAL VOC and MS COCO datasets, demonstrating significant quantitative improvements over existing baselines.

  • PASCAL VOC: In the 10-shot setting, FPD outperformed previous state-of-the-art methods by 1.5%, 4.4%, and 6.8% across three standard data splits.
  • MS COCO: FPD achieved an Average Precision (AP) of 19.3, placing it among the top-performing meta-learning methods based on the Faster R-CNN architecture.
  • Ablation Studies: The importance of fine-grained distillation was confirmed through ablation; removing the FFA module resulted in a performance drop from 68.4 to 66.3 on the VOC Set 1 benchmark.

Conclusion and Future Directions

FPD demonstrates that for few-shot tasks, local details are more critical than global averages. By breaking support images into fine-grained prototypes, the model achieves better resilience against occlusion and environmental noise. While the method shows some limitations in "ultra-low-shot" scenarios (1 or 2 shots) where data variety is nearly non-existent, it sets a new standard for FSOD. Future research may explore incorporating generative priors or self-supervised pre-training to further bolster performance when only a single support image is available.

Summary updated

BibTeX
@inproceedings{wang2024fine,
  author = {Zichen Wang and Bo Yang and Haonan Yue and Zhenghao Ma},
  title = {{Fine-Grained Prototypes Distillation for Few-Shot Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28399},
}
Figure