PETDet: Proposal Enhancement for Two-Stage Fine-Grained Object Detection

Wentao Li, Danpei Zhao, Bo Yuan, Yue Gao, Zhenwei Shi

TL;DR

This paper introduces PETDet, a novel two-stage fine-grained object detection (FGOD) framework that significantly enhances the quality of region proposals. By focusing on improving proposal generation, representation, and utilization, PETDet achieves state-of-the-art performance on challenging FGOD benchmarks while maintaining competitive inference speeds.

What Problem Does This Paper Address?

Fine-grained object detection (FGOD) goes beyond simply identifying objects; it aims to recognize specific sub-categories within broader classes (e.g., distinguishing between different models of airplanes). This is particularly challenging in aerial imagery due to varying object orientations, aspect ratios, and subtle visual differences between classes. Two-stage detectors, which first propose regions of interest (ROIs) and then classify/refine them, are common for FGOD. However, the authors argue that standard proposal generation and handling techniques, borrowed from general object detection, are suboptimal for FGOD. This leads to issues in:

  • Proposal Generation: Generating high-quality, well-localized proposals is crucial for accurate classification and avoiding redundant background information.
  • Proposal Representation: Features extracted for proposal generation and fine-grained recognition are often not sufficiently decoupled, leading to feature confusion.
  • Proposal Utilization: Existing methods often struggle to effectively utilize the best proposals, sometimes discarding valuable information due to fixed sampling ratios or non-maximum suppression (NMS) strategies.

Methods

PETDet tackles these challenges with three novel components:

  1. Quality Oriented Proposal Network (QOPN): This is an anchor-free, oriented proposal network designed to generate high-quality proposals. It incorporates dynamic label assignment (similar to ATSS) and an attention-based decomposition module (LDAM) to better separate localization and classification tasks. This aims to produce proposals that are both accurate in location and have a high probability of containing the target object.
  2. Bilinear Channel Fusion Network (BCFN): To address feature representation issues, BCFN fuses features from adjacent levels of the Feature Pyramid Network (FPN) bilinearly. This creates more independent and discriminative features for the second stage of the detector, enhancing its ability to distinguish between fine-grained categories. It draws inspiration from bilinear pooling but is adapted for channel-wise operations to maintain efficiency.
  3. Adaptive Recognition Loss (ARL): This novel loss function is designed to improve proposal utilization. Instead of relying on traditional NMS or fixed positive/negative sampling ratios, ARL dynamically re-weights proposals based on their classification score and refined IoU. This encourages the R-CNN head to focus more on high-quality proposals, leading to better fine-grained recognition learning and allowing for the discarding of NMS, which speeds up inference.

Datasets

The proposed PETDet was evaluated on several datasets:

  • FAIR1M: A large-scale, multi-class FGOD dataset for remote sensing imagery, featuring 5 coarse categories and 37 fine-grained sub-categories. Both FAIR1M-v1.0 and FAIR1M-v2.0 were used.
  • MAR20: A single-class FGOD dataset focused on military aircraft recognition.
  • ShipRSImageNet: A large-scale FGOD dataset for ship detection, with hierarchical classification.

Key Insights and Ideas

  • Proposal Quality is Paramount for FGOD: The paper strongly emphasizes that improving the region proposal stage is a more effective strategy for advancing FGOD than solely focusing on the fine-grained recognition head.
  • Standard Object Detection Techniques Aren't Optimal for FGOD: Procedures like fixed NMS and simple proposal sampling, common in general object detection, can hinder FGOD performance by either discarding good proposals or not prioritizing them effectively.
  • Bilinear Fusion for Enhanced Representation: Combining features from different FPN levels using a bilinear approach can create richer, more discriminative representations crucial for distinguishing subtle visual differences in fine-grained classes.
  • Adaptive Loss for Smarter Proposal Utilization: Dynamically weighting proposals based on their combined classification and localization quality allows the model to learn more effectively from the most informative samples, even if they are "noisy" in a traditional sense.

Key Results

PETDet achieves state-of-the-art performance on several FGOD benchmarks:

  • On FAIR1M-v2.0, PETDet with ResNet50 reached 48.81 AP, surpassing the strong baseline Oriented R-CNN by 4.91 AP.
  • It also achieved top results on FAIR1M-v1.0 (42.96 AP), MAR20 (85.91 AP), and ShipRSImageNet (74.90 AP).
  • The method demonstrates a favorable accuracy-versus-speed trade-off, with competitive inference speeds compared to other state-of-the-art methods.
  • Qualitative results show PETDet's ability to reduce false negatives and improve fine-grained category predictions, especially for challenging scenarios like densely arranged objects.

Limitations and Future Work

  • Airplane Recognition Variance: While PETDet generally outperforms baselines, confusion matrices suggest it might perform slightly worse in airplane recognition compared to the baseline on certain metrics, possibly due to how confidence scores are handled.
  • Future Work Directions: The authors suggest exploring more architecture-efficient methods, including transformer-based approaches, and investigating few-shot FGOD to address data-scarcity scenarios. They also aim to adapt the model for more practical, data-scarce applications.

Summary updated

Abstract

Fine-grained object detection (FGOD) extends object detection with the capability of fine-grained recognition. In recent two-stage FGOD methods, the region proposal serves as a crucial link between detection and fine-grained recognition. However, current methods overlook that some proposal-related procedures inherited from general detection are not equally suitable for FGOD, limiting the multitask learning from generation, representation, to utilization. In this article, we present a proposal enhancement for two-stage FGOD (PETDet) to better handle the subtasks in two-stage FGOD methods. First, an anchor-free quality-oriented proposal network (QOPN) is proposed with dynamic label assignment and attention-based decomposition to generate high-quality-oriented proposals. In addition, we present a bilinear channel fusion network (BCFN) to extract independent and discriminative features of the proposals. Furthermore, we designed a novel adaptive recognition loss (ARL) that offers guidance for the region-based convolutional neural networks (R-CNNs) head to focus on high-quality proposals. Extensive experiments validate the effectiveness of PETDet. Quantitative analysis reveals that PETDet with ResNet50 reaches state-of-the-art performance on various FGOD datasets, including FAIR1M-v1.0 (42.96 AP), FAIR1M-v2.0 (48.81 AP), MAR20 (85.91 AP), and ShipRSImageNet (74.90 AP). The proposed method also achieves superior compatibility between accuracy and inference speed. Our code and models will be released at https://github.com/canoe-Z/PETDet .
BibTeX
@article{li2024petdet,
  author = {Wentao Li and Danpei Zhao and Bo Yuan and Yue Gao and Zhenwei Shi},
  title = {{PETDet: Proposal Enhancement for Two-Stage Fine-Grained Object Detection}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  url = {https://openreview.net/forum?id=hl0YbjZHN4},
}
Figure