Feature Intertwiner for Object Detection

Hongyang Li, Bo Dai, Shaoshuai Shi, Wanli Ouyang, Xiaogang Wang

TL;DR

The paper introduces the Feature Intertwiner, a novel framework that improves object detection by using the features of large, high-resolution objects (the "reliable" set) to guide the feature learning of small, low-resolution objects (the "less reliable" set). By minimizing the distribution divergence between these two sets using Optimal Transport (OT) theory and a dynamic historical class buffer, the model learns more compact and discriminative feature representations, leading to significant performance gains—especially for challenging small objects.

What Problem Does This Paper Address?

In object detection, deep networks struggle significantly with small objects. As an image passes through convolutional layers and Region of Interest (RoI) pooling operations, fine-grained spatial details are inevitably lost due to down-sampling.

While standard approaches try to fix this using multi-scale feature pyramids (like FPN) or contextual information, they don't address the underlying structure of the feature space. Standard loss functions (like cross-entropy) do not explicitly force features of the same category at different resolutions to map close to each other. As a result, features for small objects end up scattered and disorganized in the embedding space, making them highly prone to misclassification.

Methods

The authors propose InterNet, an object detection framework equipped with a plug-and-play "Feature Intertwiner" module. The core idea is a teacher-student relationship: the well-defined features of large objects instruct the poorly-defined features of small objects on how to cluster correctly.

Figure 2: Feature intertwinter overview. Blue blobs stands for the less reliable set (small objects) and green for the reliable set (large ones). For current level l, feature map Pl of the small set is first passed into a RoI-pooling layer. Then it is fed into a make-up layer, which fuels back the information lost during RoI; it is optimized via the intertwiner module (yellow rectangle), with aid of the reliable set (green). ‘OT’ (in red) stands for the optimal transport divergence, which aligns information between levels (for details see Sec. 3.3). Pm|l is the input feature map of the reliable set for the RoI layer; m indicates higher level(s) than l.

The pipeline consists of several key components:

  • Make-up Layer: A lightweight convolutional layer placed after RoI pooling for small objects, designed to "fuel back" the details lost during down-sampling.
  • Class Buffer: Because a single training mini-batch rarely contains both large and small objects of the exact same category, the authors introduce a historical buffer. This buffer dynamically tracks and stores the running average representation (prototype) of large objects for each class.
  • Optimal Transport (OT) Regularization: To properly align the feature distributions across different network levels, the authors leverage Optimal Transport theory. Specifically, they use the Sinkhorn divergence as a differentiable loss function to minimize the distance between the feature distributions of different resolutions, forcing tighter alignment before the features hit the final classifier.

Datasets

The proposed method was trained and evaluated on two primary object detection benchmarks:

  • Microsoft COCO: Evaluated using the standard trainval35k split for training and minival / test-dev for testing.
  • PASCAL VOC 2007: Used to verify the generalization capability across different backbones (VGG-16 and ResNet-101).
  • CIFAR-10: Additionally used in a minor image classification experiment to demonstrate the broader utility of the OT alignment strategy.

Key Insights and Ideas

  • The Power of Feature Compacting: Visualizing the feature space using t-SNE reveals that without the intertwiner, features are highly scattered, leading to outliers. With the intertwiner, clusters become noticeably tighter and cleaner, drastically reducing classification errors.
t-SNE visualization of features with and without the intertwiner
  • Detaching the Buffer Stabilizes Training: The authors discovered that if the historical class buffer is updated but detached from the backpropagation gradient flow, performance improves. Treating the buffer as a stable, non-gradient-producing reference prevents training oscillations—a concept similar to replay memories in reinforcement learning.
  • The "Couch" Anomaly: In a per-category breakdown, performance dropped for the "couch" class. The authors realized that large couches in COCO are frequently covered by people, pets, and blankets. The annotations include these noisy contexts, meaning the "teacher" features for large couches were actually polluted, leading to poor guidance for smaller couches. This highlights that the intertwiner's success depends heavily on the purity of the "reliable" set.

Key Results

The Feature Intertwiner consistently outperformed baseline detectors across multiple metrics:

  • COCO Ablations: Incorporating the default intertwiner boosted overall mean Average Precision (mAP) from 32.8% to 35.2%. Crucially, the accuracy for small objects ($AP_S$) saw the largest jump, increasing by 2.6%.
  • Optimal Transport Superiority: Comparing different alignment losses showed that Optimal Transport outperformed standard $L_2$ distance and KL divergence, yielding a +2.0% AP increase over the baseline.
Ablation study showing the impact of different alignment options
  • State-of-the-Art Comparison: On the COCO test-dev set, InterNet achieved 42.5% AP (single-scale) and 44.2% AP (multi-scale), outperforming contemporary models like Mask R-CNN and RetinaNet.
  • PASCAL VOC: InterNet achieved 73.1% mAP with a VGG-16 backbone and 82.7% mAP with ResNet-101, demonstrating robust improvements over standard Faster R-CNN baselines.

Limitations and Future Work

Despite its performance gains, InterNet introduces some overhead:

  • Computational Complexity: Training takes roughly 3.4 days on 8 GPUs, which is slower than standard Mask R-CNN.
  • Memory and Inference Overhead: Memory consumption increases from 8.3 GB to 9.6 GB per GPU during training, and inference time increases by about 5% (from 308ms to 325ms per image).
  • Context Dependency: As seen with the "couch" example, if the large objects are heavily occluded or visually noisy, the buffer stores inaccurate prototypes, which can degrade performance for that specific class.

For future work, the authors suggest extending the feature intertwiner concept beyond object detection to other domains, such as general data classification, provided a logical division between "reliable" and "less reliable" subsets can be established.

Summary updated

Abstract

A well-trained model should classify objects with a unanimous score for every category. This requires the high-level semantic features should be as much alike as possible among samples. To achive this, previous works focus on re-designing the loss or proposing new regularization constraints. In this paper, we provide a new perspective. For each category, it is assumed that there are two feature sets: one with reliable information and the other with less reliable source. We argue that the reliable set could guide the feature learning of the less reliable set during training - in spirit of student mimicking teacher behavior and thus pushing towards a more compact class centroid in the feature space. Such a scheme also benefits the reliable set since samples become closer within the same category - implying that it is easier for the classifier to identify. We refer to this mutual learning process as feature intertwiner and embed it into object detection. It is well-known that objects of low resolution are more difficult to detect due to the loss of detailed information during network forward pass (e.g., RoI operation). We thus regard objects of high resolution as the reliable set and objects of low resolution as the less reliable set. Specifically, an intertwiner is designed to minimize the distribution divergence between two sets. The choice of generating an effective feature representation for the reliable set is further investigated, where we introduce the optimal transport (OT) theory into the framework. Samples in the less reliable set are better aligned with aid of OT metric. Incorporated with such a plug-and-play intertwiner, we achieve an evident improvement over previous state-of-the-arts.
BibTeX
@inproceedings{li2019feature,
  author = {Hongyang Li and Bo Dai and Shaoshuai Shi and Wanli Ouyang and Xiaogang Wang},
  title = {{Feature Intertwiner for Object Detection}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure