Better to Follow, Follow to Be Better: Towards Precise Supervision of Feature Super-Resolution for Small Object Detection

Junhyug Noh, Wonho Bae, Wonhee Lee, Jinhwan Seo, Gunhee Kim

Introduction

Small object detection remains a significant challenge in computer vision. While state-of-the-art detectors like Faster R-CNN perform exceptionally well on standard-sized objects, their accuracy drops sharply when objects occupy only a few pixels. This performance gap stems from two primary issues: information loss during downsampling and feature distortion caused by misalignment during pooling operations.

The Challenge of Precise Supervision

A common strategy to combat information loss is Super-Resolution (SR), which aims to recover high-resolution (HR) details from low-resolution (LR) inputs. However, applying SR to object detection features is non-trivial. Previous methods often relied on indirect supervision from detection losses or used naive HR features as targets.

The authors identify a critical "Relative Receptive Field (RRF) mismatch" in these approaches. A small object in an LR image inherently captures more surrounding context relative to its size than a large object in an HR image. Forcing an LR feature to mimic an HR feature without accounting for this difference leads to imprecise supervision and suboptimal performance.

Comparison of SR targets: Naive vs. Proposed. Receptive field alignment is crucial for effective supervision.

Proposed Methodology: Precise Feature Super-Resolution

To address these challenges, the authors propose a GAN-based framework integrated into the Faster R-CNN pipeline. The architecture is built around several key components:

SR Target Extractor

The core innovation is the SR Target Extractor, which serves as the "teacher" network. By utilizing atrous (dilated) convolutions, it ensures that the high-resolution target features encompass the same relative context as the low-resolution input features. This alignment provides a precise supervision signal that resolves the RRF mismatch.

SR Feature Generator and Adversarial Training

The SR Feature Generator is a GAN-based module that upscales LR features. It doesn't just upscale the final layer; it incorporates "sub-layer" features from earlier in the backbone to provide fine-grained structural blueprints. A discriminator is employed to distinguish between generated SR features and real features from the Target Extractor, driving the generator toward high-fidelity outputs.

The proposed architecture featuring the SR Target Extractor, GAN-based generator, and dual predictor structure.

Dual Predictors

The model employs a split workload strategy. Large objects are processed by a standard "Large Predictor," while small objects are enhanced by the SR Generator before being handled by a specialized "Small Predictor." This ensures that the model can optimize for different scales without compromise.

Experimental Results

The framework was evaluated on three major benchmarks: Tsinghua-Tencent 100K (TT100K), PASCAL VOC 2007, and MS COCO.

  • TT100K: On this traffic sign dataset, where objects are extremely small, the model achieved a massive improvement. Using a ResNet-101 backbone, the F1 score for small objects jumped from 75.2 to 84.3.
  • MS COCO: The approach demonstrated consistent gains across all object sizes, proving that improving small-object features does not negatively impact the detection of larger objects.
  • Ablation Studies: The results confirmed that without the atrous convolutions in the Target Extractor (i.e., using naive targets), the SR process could actually degrade performance, highlighting the importance of precise supervision.

Conclusion

This work demonstrates that effective feature super-resolution for small object detection requires more than just upscaling; it requires precise alignment of receptive fields. By introducing the SR Target Extractor with atrous convolutions, the authors provide a robust framework for supervising the generation of high-fidelity features, leading to state-of-the-art results on challenging datasets.

Summary updated

Abstract

In spite of recent success of proposal-based CNN models for object detection, it is still difficult to detect small objects due to the limited and distorted information that small region of interests (RoI) contain. One way to alleviate this issue is to enhance the features of small RoIs using a super-resolution (SR) technique. We investigate how to improve feature-level super-resolution especially for small object detection, and discover its performance can be significantly improved by (i) utilizing proper high-resolution target features as supervision signals for training of a SR model and (ii) matching the relative receptive fields of training pairs of input low-resolution features and target high-resolution features. We propose a novel feature-level super-resolution approach that not only correctly addresses these two desiderata but also is integrable with any proposal-based detectors with feature pooling. In our experiments, our approach significantly improves the performance of Faster R-CNN on three benchmarks of Tsinghua-Tencent 100K, PASCAL VOC and MS COCO. The improvement for small objects is remarkably large, and encouragingly, those for medium and large objects are nontrivial too. As a result, we achieve new state-of-the-art performance on Tsinghua-Tencent 100K and highly competitive results on both PASCAL VOC and MS COCO.
BibTeX
@inproceedings{noh2019better,
  author = {Junhyug Noh and Wonho Bae and Wonhee Lee and Jinhwan Seo and Gunhee Kim},
  title = {{Better to Follow, Follow to Be Better: Towards Precise Supervision of Feature Super-Resolution for Small Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00982},
}
Figure