SAUI: Scale-Aware Unseen Imagineer for Zero-Shot Object Detection

Jiahao Wang, Caixia Yan, Weizhan Zhang, Huan Liu, Hao Sun, Qinghua Zheng

Introduction: The Scale-Blindness Problem in ZSD

Zero-Shot Object Detection (ZSD) aims to detect objects from classes not seen during training by leveraging semantic descriptions. A common strategy involves synthesizing visual features for these unseen classes. However, existing "imagineers" often suffer from scale-blindness, generating features at a single, fixed scale. This leads to "monotonous imaginations" that fail to account for the significant size variations objects exhibit in real-world scenes due to distance or perspective.

Conceptual comparison between traditional ZSD frameworks and the scale-aware SAUI approach.

The SAUI Framework

To address this, the Scale-Aware Unseen Imagineer (SAUI) introduces a multi-scale approach to feature synthesis. The framework is built upon three core pillars:

1. Coarse-to-Fine Feature Extractor

Unlike standard backbones that might output a single feature map, SAUI utilizes a coarse-to-fine extractor. It aggregates information from multiple levels of a Feature Pyramid Network (FPN), capturing both the broad outlines and the intricate details of objects.

2. Series-GAN Synthesizer

The heart of SAUI is the Series-GAN, a chain of generators ($G_1$ to $G_4$) designed to produce multi-scale features. By treating feature generation as a sequence-to-sequence task, the model progressively refines its "imagination" from coarse representations to high-resolution details.

The SAUI architecture, highlighting the three-phase training process and the Series-GAN synthesizer.

3. Multi-scale Contrastive Imagination

To ensure the synthesized features are both diverse and representative, SAUI employs a three-pronged contrastive loss:

  • Intra-scale Outline Maintaining ($L_{main}$): Preserves class separability at each scale.
  • Intra-scale Detail Diverging ($L_{div}$): Encourages variety within a class to prevent mode collapse.
  • Inter-scale Mutual Constructing ($L_{con}$): Ensures consistency across different scales for the same object concept.

Experimental Evaluation

SAUI was evaluated on MS COCO, DIOR (remote sensing), and PASCAL VOC.

On MS COCO, SAUI achieved 22.1% mAP, significantly outperforming previous methods. Most notably, it nearly tripled the performance on small objects ($AP_s$), demonstrating the effectiveness of scale-aware synthesis.

On the DIOR dataset, which features extreme scale variations, SAUI reached 16.6% mAP—a 42% improvement over previous state-of-the-art results. This confirms the framework's robustness in challenging, multi-scale environments.

Conclusion

SAUI demonstrates that scale is a critical dimension in Zero-Shot Object Detection. By explicitly modeling scale diversity through a coarse-to-fine architecture and a specialized GAN synthesizer, SAUI overcomes the limitations of "monotonous imaginations," particularly for small and scale-varying objects. While performance gains are less pronounced on datasets with centered, large objects like PASCAL VOC, SAUI sets a new benchmark for complex, real-world detection tasks.

Summary updated

BibTeX
@inproceedings{wang2024saui,
  author = {Jiahao Wang and Caixia Yan and Weizhan Zhang and Huan Liu and Hao Sun and Qinghua Zheng},
  title = {{SAUI: Scale-Aware Unseen Imagineer for Zero-Shot Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28353},
}
Figure