Few-Shot Object Detection in Remote Sensing: Lifting the Curse of Incompletely Annotated Novel Objects

Fahong Zhang, Yilei Shi, Zhitong Xiong, Xiao Xiang Zhu

TL;DR

This paper tackles the challenge of "incompletely annotated novel objects" (IANO) in few-shot object detection (FSOD) for remote sensing. They propose a self-training approach (ST-FSOD) that uses a teacher-student framework to identify and leverage unlabeled novel objects during training, significantly improving detection performance.

What Problem Does This Paper Address?

Object detection is crucial for analyzing satellite imagery, but training robust models requires vast amounts of labeled data. Few-shot object detection (FSOD) aims to address this by enabling models to detect new object classes with only a few labeled examples. However, a critical issue arises in FSOD when an image contains multiple instances of a novel object, but only a subset are annotated. This "incompletely annotated novel objects" (IANO) problem causes standard FSOD models to treat unannotated instances as background, leading to confusion and hindering their ability to learn novel classes effectively. This is particularly problematic in remote sensing due to smaller object sizes and complex scenes.

Methods

The authors propose a self-training-based FSOD (ST-FSOD) approach built upon the Faster R-CNN framework. The core idea is to use a "teacher-student" mechanism to generate "pseudo-labels" for unlabeled novel objects.

  • Self-Training Region Proposal Network (ST-RPN): This module separates the extraction of proposals for base and novel objects. It uses a teacher RPN to identify potentially novel objects and a student RPN, trained with pseudo-labels, to generate novel proposals. This separation prevents the fine-tuning for novel classes from negatively impacting the detection of base classes.
  • Self-Training Bounding Box Head (ST-BBH): This module refines the bounding box predictions. It uses a teacher BBH to provide high-confidence pseudo-labels for unlabeled novel objects, which are then used to train the student BBH. This helps the model recall more novel objects that might otherwise be missed.
  • Balanced Sampling Strategy (BSS): To ensure a more balanced training set, the authors include all base class images and annotations and increase the sampling probability of images containing novel objects.

Datasets

The proposed method was evaluated on three large-scale public remote sensing datasets:

  • NWPU-VHR10 v2: Contains 1,172 images with 10 object categories.
  • DIOR: Features 23,463 images with 20 object categories.
  • iSAID: A large-scale instance segmentation dataset with 2,806 images and 15 object categories.

The paper follows established split strategies for base and novel classes from previous works.

Key Insights and Ideas

  • IANO is a critical bottleneck in remote sensing FSOD: The paper highlights that the common practice of treating unannotated objects as background is a significant problem, especially in complex remote sensing scenes.
  • Self-training is a powerful tool for IANO: By leveraging pseudo-labels generated from a teacher model, self-training can effectively identify and utilize unlabeled novel objects, improving recall.
  • Separating base and novel proposal extraction is beneficial: This prevents the fine-tuning process for novel classes from degrading the performance on well-established base classes.
  • Balanced sampling is crucial for base class performance: The proposed BSS strategy helps maintain strong performance on base classes during few-shot fine-tuning.

Key Results

The ST-FSOD method achieved state-of-the-art performance across various FSOD settings on all three datasets.

  • NWPU-VHR10 v2: Outperformed existing methods by a notable margin (3-7%) in 3 and 20-shot settings.
  • DIOR: Consistently outperformed the second-best approach by 5-7%, with improvements exceeding 10% in some splits.
  • iSAID: Demonstrated significant enhancements ranging from 5% to over 10%.
  • Qualitative results: Visualizations showed the method's effectiveness in detecting small and previously missed novel objects, particularly ships, which are often challenging in remote sensing imagery.

Limitations and Future Work

  • Base class performance degradation: While the proposed method significantly improves novel class detection, there's a slight decrease in base class performance. The authors suggest future work could focus on designing generalized FSOD methods that prevent forgetting previously learned base knowledge.
  • Computational cost: The ST-RPN module introduces some additional computational overhead during training, though it's noted that FSOD training involves fewer iterations. Optimization through CUDA implementations is suggested.
  • Failure cases: Missed detections were observed for very small objects, objects with large size variance, and objects lacking clear boundaries. The authors suggest that existing techniques for general object detection could be adapted to improve these aspects.

Summary updated

Abstract

Object detection is an essential and fundamental task in computer vision and satellite image processing. Existing deep learning methods have achieved impressive performance thanks to the availability of large-scale annotated datasets. Yet, in real-world applications the availability of labels is limited. In this context, few-shot object detection (FSOD) has emerged as a promising direction, which aims at enabling the model to detect novel objects with only few of them annotated. However, many existing FSOD algorithms overlook a critical issue: when an input image contains multiple novel objects and only a subset of them are annotated, the unlabeled objects will be considered as background during training. This can cause confusions and severely impact the model's ability to recall novel objects. To address this issue, we propose a self-training-based FSOD (ST-FSOD) approach, which incorporates the self-training mechanism into the few-shot fine-tuning process. ST-FSOD aims to enable the discovery of novel objects that are not annotated, and take them into account during training. On the one hand, we devise a two-branch region proposal networks (RPN) to separate the proposal extraction of base and novel objects, On another hand, we incorporate the student-teacher mechanism into RPN and the region of interest (RoI) head to include those highly confident yet unlabeled targets as pseudo labels. Experimental results demonstrate that our proposed method outperforms the state-of-the-art in various FSOD settings by a large margin. The codes will be publicly available at https://github.com/zhu-xlab/ST-FSOD.
BibTeX
@article{zhang2024few,
  author = {Fahong Zhang and Yilei Shi and Zhitong Xiong and Xiao Xiang Zhu},
  title = {{Few-Shot Object Detection in Remote Sensing: Lifting the Curse of Incompletely Annotated Novel Objects}},
  journal = {IEEE TGRS},
  year = {2024},
  doi = {10.1109/tgrs.2023.3347329},
}
Figure