TL;DR
This paper tackles the challenge of "incompletely annotated novel objects" (IANO) in few-shot object detection (FSOD) for remote sensing. They propose a self-training approach (ST-FSOD) that uses a teacher-student framework to identify and leverage unlabeled novel objects during training, significantly improving detection performance.
What Problem Does This Paper Address?
Object detection is crucial for analyzing satellite imagery, but training robust models requires vast amounts of labeled data. Few-shot object detection (FSOD) aims to address this by enabling models to detect new object classes with only a few labeled examples. However, a critical issue arises in FSOD when an image contains multiple instances of a novel object, but only a subset are annotated. This "incompletely annotated novel objects" (IANO) problem causes standard FSOD models to treat unannotated instances as background, leading to confusion and hindering their ability to learn novel classes effectively. This is particularly problematic in remote sensing due to smaller object sizes and complex scenes.
Methods
The authors propose a self-training-based FSOD (ST-FSOD) approach built upon the Faster R-CNN framework. The core idea is to use a "teacher-student" mechanism to generate "pseudo-labels" for unlabeled novel objects.
- Self-Training Region Proposal Network (ST-RPN): This module separates the extraction of proposals for base and novel objects. It uses a teacher RPN to identify potentially novel objects and a student RPN, trained with pseudo-labels, to generate novel proposals. This separation prevents the fine-tuning for novel classes from negatively impacting the detection of base classes.
- Self-Training Bounding Box Head (ST-BBH): This module refines the bounding box predictions. It uses a teacher BBH to provide high-confidence pseudo-labels for unlabeled novel objects, which are then used to train the student BBH. This helps the model recall more novel objects that might otherwise be missed.
- Balanced Sampling Strategy (BSS): To ensure a more balanced training set, the authors include all base class images and annotations and increase the sampling probability of images containing novel objects.
Datasets
The proposed method was evaluated on three large-scale public remote sensing datasets:
- NWPU-VHR10 v2: Contains 1,172 images with 10 object categories.
- DIOR: Features 23,463 images with 20 object categories.
- iSAID: A large-scale instance segmentation dataset with 2,806 images and 15 object categories.
The paper follows established split strategies for base and novel classes from previous works.
Key Insights and Ideas
- IANO is a critical bottleneck in remote sensing FSOD: The paper highlights that the common practice of treating unannotated objects as background is a significant problem, especially in complex remote sensing scenes.
- Self-training is a powerful tool for IANO: By leveraging pseudo-labels generated from a teacher model, self-training can effectively identify and utilize unlabeled novel objects, improving recall.
- Separating base and novel proposal extraction is beneficial: This prevents the fine-tuning process for novel classes from degrading the performance on well-established base classes.
- Balanced sampling is crucial for base class performance: The proposed BSS strategy helps maintain strong performance on base classes during few-shot fine-tuning.
Key Results
The ST-FSOD method achieved state-of-the-art performance across various FSOD settings on all three datasets.
- NWPU-VHR10 v2: Outperformed existing methods by a notable margin (3-7%) in 3 and 20-shot settings.
- DIOR: Consistently outperformed the second-best approach by 5-7%, with improvements exceeding 10% in some splits.
- iSAID: Demonstrated significant enhancements ranging from 5% to over 10%.
- Qualitative results: Visualizations showed the method's effectiveness in detecting small and previously missed novel objects, particularly ships, which are often challenging in remote sensing imagery.
Limitations and Future Work
- Base class performance degradation: While the proposed method significantly improves novel class detection, there's a slight decrease in base class performance. The authors suggest future work could focus on designing generalized FSOD methods that prevent forgetting previously learned base knowledge.
- Computational cost: The ST-RPN module introduces some additional computational overhead during training, though it's noted that FSOD training involves fewer iterations. Optimization through CUDA implementations is suggested.
- Failure cases: Missed detections were observed for very small objects, objects with large size variance, and objects lacking clear boundaries. The authors suggest that existing techniques for general object detection could be adapted to improve these aspects.
Summary updated