Extended Feature Pyramid Network for Small Object Detection

Chunfang Deng, Mengmeng Wang, Liang Liu, Yong Liu

TL;DR

This paper introduces the Extended Feature Pyramid Network (EFPN) to tackle the persistent challenge of detecting small objects in images. EFPN enhances existing feature pyramid networks by adding a dedicated high-resolution level for small objects, a novel "feature texture transfer" module for richer detail extraction, and a balanced loss function to address foreground-background imbalances, achieving state-of-the-art results.

What Problem Does This Paper Address?

Detecting small objects in images is a notoriously difficult problem in computer vision. These objects, often comprising only a few pixels, have their crucial visual information easily lost during the downsampling operations inherent in Convolutional Neural Networks (CNNs). While Feature Pyramid Networks (FPNs) have been a significant step forward by leveraging features from different network layers to handle scale variations, they still struggle with small objects. The paper identifies that feature coupling across scales in standard FPNs can still lead to confusion and degraded performance for these tiny targets. Existing super-resolution (SR) techniques applied to object detection often rely on fabricating details or are computationally expensive, leading to potential false positives or performance bottlenecks.

Methods

The core of this work is the Extended Feature Pyramid Network (EFPN). EFPN builds upon the standard FPN architecture by introducing an additional, high-resolution pyramid level specifically designed for small object detection.

Key novelties include:

  • Feature Texture Transfer (FTT) Module: This module is designed to enrich the features of the extended pyramid level. It takes a low-resolution feature map (P3) and a high-resolution feature map (P2) as input. It uses a "content extractor" to process P3 and a "texture extractor" to leverage P2. The FTT module then intelligently combines semantic information from P3 with credible regional textures from P2, effectively super-resolving features and extracting fine-grained details without fabricating them. This is inspired by reference-based super-resolution techniques.
  • Foreground-Background-Balanced Loss Function: To address the significant imbalance between the number of foreground (object) pixels and background pixels, especially critical for small objects that occupy tiny image areas, the authors propose a balanced loss. This loss comprises two parts: a global reconstruction loss (Lglob) and a positive patch loss (Lpos). The positive patch loss specifically focuses on improving the feature quality of foreground areas, drawing more attention to the small objects.

The EFPN architecture integrates these components by first constructing the standard FPN layers and then incorporating the FTT module to generate the extended, high-resolution feature map (P2'). This extended pyramid, along with the standard FPN levels, is then fed to the object detector for localization and classification.

Datasets

The proposed EFPN was evaluated on two datasets:

  • Tsinghua-Tencent 100K: This dataset is specifically designed for traffic-sign detection and classification. It's highlighted as an excellent benchmark for small object detection due to the fact that 92% of its instances cover less than 0.2% of the image area. The dataset is divided into three scale categories: small (< 32² pixels), medium (32² < area < 96² pixels), and large (> 96² pixels).
  • MS COCO: A widely-used large-scale dataset for general object detection. The authors focus on the "small category" (area < 32² pixels) of MS COCO to demonstrate EFPN's effectiveness on general object detection tasks.

Key Insights and Ideas

  • Feature Coupling is a Bottleneck: The paper strongly argues that even with FPNs, the way features from different scales are combined can still hinder small object detection. The proposed EFPN aims to decouple these scales more effectively.
  • Credible Details Matter More Than Fabrication: The FTT module's success suggests that extracting real regional details from higher-resolution features is more beneficial for small object detection than simply trying to hallucinate details based on low-resolution information alone, which can lead to artifacts.
  • Loss Function Imbalance is Critical: The foreground-background-balanced loss function is shown to be crucial for improving performance, particularly for small objects that are easily overwhelmed by background signals.
  • Efficiency is Achieved: The EFPN design manages to improve performance without a prohibitive increase in computational cost or memory usage, making it practical for real-world applications.

Key Results

  • State-of-the-Art Performance: EFPN achieves state-of-the-art results on both the Tsinghua-Tencent 100K dataset and for the small object category in MS COCO.
  • Tsinghua-Tencent 100K: Compared to Faster R-CNN with ResNeXt-101-FPN, single-scale EFPN shows a significant improvement in small object accuracy (3.4%) and recall (0.2%). The multi-scale EFPN achieves an F1 score of 95.7 for large objects, 96.2 for medium, and 95.7 for small objects, outperforming other methods.
  • MS COCO: EFPN demonstrates prominent gains on ResNet-50/ResNeXt-101 backbones, outperforming other FPN variants and SR-based methods. For instance, with ResNeXt-101, EFPN achieves an APS of 26.7 and ARS of 42.3, surpassing previous state-of-the-art.
  • Efficiency: EFPN achieves comparable performance to multi-scale FPN tests but with significantly lower runtime and GPU memory usage, making it more efficient.

Limitations and Future Work

  • Extended Pyramid Level Alone is Insufficient: Ablation studies show that simply adding an extended pyramid level without the FTT module and balanced loss function provides only a limited improvement. This highlights the importance of the proposed FTT module and loss function.
  • Future Work: The authors suggest that EFPN can be combined with various detectors to enhance small object detection in more specific scenarios like face detection or satellite image detection. They express interest in exploring these applications further.

Summary updated

Abstract

Small object detection remains an unsolved challenge because it is hard to extract information of small objects with only a few pixels. While scale-level corresponding detection in feature pyramid network alleviates this problem, we find feature coupling of various scales still impairs the performance of small objects. In this paper, we propose extended feature pyramid network (EFPN) with an extra high-resolution pyramid level specialized for small object detection. Specifically, we design a novel module, named feature texture transfer (FTT), which is used to super-resolve features and extract credible regional details simultaneously. Moreover, we design a foreground-background-balanced loss function to alleviate area imbalance of foreground and background. In our experiments, the proposed EFPN is efficient on both computation and memory, and yields state-of-the-art results on small traffic-sign dataset Tsinghua-Tencent 100K and small category of general object detection dataset MS COCO.
BibTeX
@article{deng2022extended,
  author = {Chunfang Deng and Mengmeng Wang and Liang Liu and Yong Liu},
  title = {{Extended Feature Pyramid Network for Small Object Detection}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3074273},
}
Figure