Feature Intertwiner for Object Detection

Hongyang Li, Bo Dai, Shaoshuai Shi, Wanli Ouyang, Xiaogang Wang

In object detection, size matters. Large objects provide rich textures and clear shapes, making them easy for neural networks to classify. Small objects, however, lose critical details during a network's downsampling and pooling processes. This leads to a "feature gap" where a large car and a tiny car of the same model look very different in the high-dimensional feature space.

While previous works have tried to fix this using multi-scale image pyramids or specialized loss functions, they often fail to ensure that features within the same category remain "compact" (similar to each other) regardless of resolution. To bridge this gap, researchers have introduced Feature Intertwiner (or InterNet), a novel framework that improves object detection by treating large objects as "teachers" that guide the feature learning of small "student" objects.

The Feature Gap in Object Detection

The core problem InterNet addresses is that "unreliable" samples (like small, low-resolution objects) do not naturally align with "reliable" samples (large, high-resolution objects) in the feature space.

t-SNE visualization of feature distributions before (a) and after (b) applying the Feature Intertwiner.

By forcing the features of unreliable samples to match the features of reliable ones, the model creates a more compact and robust representation for every category, making it significantly easier for the final classifier to do its job.

How the Feature Intertwiner Works

The authors introduce a plug-and-play module designed around a mutual learning process between a high-resolution reliable set and a low-resolution less reliable set.

Architecture of the Feature Intertwiner module, highlighting the Make-up layer, Critic, and Optimal Transport alignment.

Key components of this architecture include:

  • The Make-up Layer: A small convolutional block designed to "fuel back" lost information into the features of small objects.
  • Class Buffer: In a single training mini-batch, you might have a small dog but no large dog to compare it to. The authors solve this by maintaining a historical buffer that stores a "prototype" (running average) of features for large instances of every category.
  • Optimal Transport (OT) Alignment: To make the features of small objects match the reliable ones, the authors use the Sinkhorn divergence (a computationally efficient version of Optimal Transport). Unlike standard metrics like KL divergence, OT is better at measuring distances between distributions in high-dimensional spaces, providing a smoother "supervision signal" for the network to follow.
  • The Critic Module: A small sub-network that extracts the specific semantic information needed to compare the two sets, ignoring spatial noise.

Key Technical Insights

  1. Teacher-Student Synergy: The "intertwining" isn't just one-way. While the reliable set guides the less reliable one, the process forces all samples of a category toward a more compact centroid. This actually helps the classifier identify the "reliable" samples more easily, too.
  2. The "Relative" Nature of Scale: "Large" and "small" are relative concepts. A proposal that is "large" at one level of a Feature Pyramid Network (FPN) might be "small" at the next. The level-agnostic buffer captures the best possible version of an object to act as the ultimate reference.
  3. Detaching the Buffer: A crucial technical finding was that performance improved when the buffer was "detached" from back-propagation. Treating the buffer as a stable target (similar to a target network in Reinforcement Learning) helped the model converge more effectively.

Results and Performance

The researchers evaluated InterNet on two primary benchmarks: MS COCO (using the trainval35k split) and PASCAL VOC 2007. The Feature Intertwiner provided a consistent boost across the board:

  • COCO Improvements: On a ResNet-50-FPN baseline, the mAP (mean Average Precision) increased from 32.8% to 35.2%.
  • Small Object Mastery: The most significant gain was in small object detection ($AP_S$), which saw a massive improvement of 2.6%.
  • State-of-the-Art: When combined with a ResNet-101 backbone and multi-scale testing, InterNet achieved 44.2% mAP on COCO, outperforming contemporary models like Mask R-CNN and RetinaNet.
  • Efficiency: The module is lightweight, adding only about 5% to the inference time (increasing from 308ms to 325ms per image on a Titan Pascal X).

Limitations and Future Work

Interestingly, the authors noted a curious failure case: the "couch" category. Performance actually dropped for couches because large couches in the COCO dataset are often cluttered with people, pets, and pillows. This "noise" in the reliable set provided poor guidance for smaller, cleaner couch samples.

Future directions for this research include refining how the "reliable set" is selected to avoid this "noisy teacher" problem, exploring more sophisticated ways to update the class buffer beyond simple running averages, and applying the intertwiner concept to other domains like general data classification or cross-domain adaptation.

Summary updated

Abstract

A well-trained model should classify objects with unanimous score for every category. This requires the high-level semantic features should be alike among samples, despite a wide span in resolution, texture, deformation, etc. Previous works focus on re-designing the loss function or proposing new regularization constraints on the loss. In this paper, we address this problem via a new perspective. For each category, it is assumed that there are two sets in the feature space: one with more reliable information and the other with less reliable source. We argue that the reliable set could guide the feature learning of the less reliable set during training - in spirit of student mimicking teacher’s behavior and thus pushing towards a more compact class centroid in the high-dimensional space. Such a scheme also benefits the reliable set since samples become more closer within the same category - implying that it is easilier for the classifier to identify. We refer to this mutual learning process as feature intertwiner and embed the spirit into object detection. It is well-known that objects of low resolution are more difficult to detect due to the loss of detailed information during network forward pass. We thus regard objects of high resolution as the reliable set and objects of low resolution as the less reliable set. Specifically, an intertwiner is achieved by minimizing the distribution divergence between two sets. We design a historical buffer to represent all previous samples in the reliable set and utilize them to guide the feature learning of the less reliable set. The design of obtaining an effective feature representation for the reliable set is further investigated, where we introduce the optimal transport (OT) algorithm into the framework. Samples in the less reliable set are better aligned with the reliable set with aid of OT metric. Incorporated with such a plug-and-play intertwiner, we achieve an evident improvement over previous state-of-the-arts on the COCO object detection benchmark.
BibTeX
@inproceedings{li2019feature,
  author = {Hongyang Li and Bo Dai and Shaoshuai Shi and Wanli Ouyang and Xiaogang Wang},
  title = {{Feature Intertwiner for Object Detection}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure