Breaking Immutable: Information-Coupled Prototype Elaboration for Few-Shot Object Detection

Xiaonan Lu, Wenhui Diao, Yongqiang Mao, Junxi Li, Peijin Wang, Xian Sun, Kun FuAAAI 2023 · DOI

Introduction

Few-Shot Object Detection (FSOD) is a challenging task that requires models to detect novel objects using only a handful of labeled examples, known as the "support set." Most current meta-learning approaches rely on extracting a "prototype"—a feature vector representing a class—from support images and matching it against features in a "query image."

However, these methods face two significant hurdles. First, prototypes are typically "immutable," meaning they are static representations extracted solely from support images without considering the specific context (lighting, perspective, or background) of the query image. Second, standard prototype generation often uses simple global average pooling, which treats all pixels equally. This allows cluttered backgrounds to overwhelm the actual object features, leading to diluted and unrepresentative prototypes.

Comparison between previous immutable prototype methods and the proposed dynamic ICPE approach.

The ICPE Framework

To address these limitations, this research introduces Information-Coupled Prototype Elaboration (ICPE). Built upon the Meta R-CNN architecture, ICPE transforms the static prototype into a dynamic, query-aware representation through two primary modules: Conditional Information Coupling (CIC) and Prototype Dynamic Aggregation (PDA).

Overview of the ICPE architecture, highlighting the Query and Support branches coupled via the CIC and PDA modules.

Conditional Information Coupling (CIC)

The CIC module is designed to break the "immutability" of prototypes. It employs a non-local attention mechanism to "pollinate" support features with relevant information from the query image. To prevent the introduction of redundant query noise, a Conditional Coupling Mechanism based on cosine similarity ensures that only query information relevant to the support class is integrated. This process makes the resulting prototypes contextually aware of the specific image being searched.

Prototype Dynamic Aggregation (PDA)

Rather than relying on simple averaging, the PDA module uses weighted aggregation to highlight salient object features and suppress background noise. It consists of two sub-mechanisms:

  • Intra-image Dynamic Aggregation (Intra-DAM): This mechanism calculates the importance of each pixel within a single support image, assigning higher weights to the object of interest and lower weights to the background.
  • Inter-image Dynamic Aggregation (Inter-DAM): In multi-shot settings (e.g., 5-shot or 10-shot), this mechanism identifies which support images are most similar to the current query image and weights their contributions more heavily in the final class prototype.

Experimental Results

The ICPE model was evaluated on the Pascal VOC and MS COCO benchmarks, achieving state-of-the-art (SOTA) performance across nearly all settings.

On the Pascal VOC dataset, ICPE reached 54.3% mAP in the 1-shot setting, a massive improvement over the Meta R-CNN baseline (19.9%) and surpassing other SOTA methods like DeFRCN (53.6%). On the more complex MS COCO dataset, ICPE improved Average Precision (AP) by over 10% compared to the baseline, reaching 19.3% and 23.1% AP in 10 and 30-shot settings, respectively.

Ablation studies confirmed the necessity of both modules: removing the CIC module resulted in a 5.2% performance drop, while removing the PDA module led to a 3.7% decrease.

Conclusion

The core insight of ICPE is that prototypes should not be abstract class representations but should be dynamic and query-aware. By coupling information between branches and using smart aggregation, the model effectively preserves the object "signal" against background noise. While the framework introduces some computational overhead—reducing processing speed from 13.4 FPS to 9.6 FPS—the substantial gains in detection accuracy make it a powerful advancement in few-shot learning.

Summary updated

BibTeX
@inproceedings{lu2023breaking,
  author = {Xiaonan Lu and Wenhui Diao and Yongqiang Mao and Junxi Li and Peijin Wang and Xian Sun and Kun Fu},
  title = {{Breaking Immutable: Information-Coupled Prototype Elaboration for Few-Shot Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25274},
}
Figure