TL;DR
The paper introduces the Feature Intertwiner, a novel framework that improves object detection by using the features of large, high-resolution objects (the "reliable" set) to guide the feature learning of small, low-resolution objects (the "less reliable" set). By minimizing the distribution divergence between these two sets using Optimal Transport (OT) theory and a dynamic historical class buffer, the model learns more compact and discriminative feature representations, leading to significant performance gains—especially for challenging small objects.
What Problem Does This Paper Address?
In object detection, deep networks struggle significantly with small objects. As an image passes through convolutional layers and Region of Interest (RoI) pooling operations, fine-grained spatial details are inevitably lost due to down-sampling.
While standard approaches try to fix this using multi-scale feature pyramids (like FPN) or contextual information, they don't address the underlying structure of the feature space. Standard loss functions (like cross-entropy) do not explicitly force features of the same category at different resolutions to map close to each other. As a result, features for small objects end up scattered and disorganized in the embedding space, making them highly prone to misclassification.
Methods
The authors propose InterNet, an object detection framework equipped with a plug-and-play "Feature Intertwiner" module. The core idea is a teacher-student relationship: the well-defined features of large objects instruct the poorly-defined features of small objects on how to cluster correctly.
The pipeline consists of several key components:
- Make-up Layer: A lightweight convolutional layer placed after RoI pooling for small objects, designed to "fuel back" the details lost during down-sampling.
- Class Buffer: Because a single training mini-batch rarely contains both large and small objects of the exact same category, the authors introduce a historical buffer. This buffer dynamically tracks and stores the running average representation (prototype) of large objects for each class.
- Optimal Transport (OT) Regularization: To properly align the feature distributions across different network levels, the authors leverage Optimal Transport theory. Specifically, they use the Sinkhorn divergence as a differentiable loss function to minimize the distance between the feature distributions of different resolutions, forcing tighter alignment before the features hit the final classifier.
Datasets
The proposed method was trained and evaluated on two primary object detection benchmarks:
- Microsoft COCO: Evaluated using the standard
trainval35ksplit for training andminival/test-devfor testing. - PASCAL VOC 2007: Used to verify the generalization capability across different backbones (VGG-16 and ResNet-101).
- CIFAR-10: Additionally used in a minor image classification experiment to demonstrate the broader utility of the OT alignment strategy.
Key Insights and Ideas
- The Power of Feature Compacting: Visualizing the feature space using t-SNE reveals that without the intertwiner, features are highly scattered, leading to outliers. With the intertwiner, clusters become noticeably tighter and cleaner, drastically reducing classification errors.
- Detaching the Buffer Stabilizes Training: The authors discovered that if the historical class buffer is updated but detached from the backpropagation gradient flow, performance improves. Treating the buffer as a stable, non-gradient-producing reference prevents training oscillations—a concept similar to replay memories in reinforcement learning.
- The "Couch" Anomaly: In a per-category breakdown, performance dropped for the "couch" class. The authors realized that large couches in COCO are frequently covered by people, pets, and blankets. The annotations include these noisy contexts, meaning the "teacher" features for large couches were actually polluted, leading to poor guidance for smaller couches. This highlights that the intertwiner's success depends heavily on the purity of the "reliable" set.
Key Results
The Feature Intertwiner consistently outperformed baseline detectors across multiple metrics:
- COCO Ablations: Incorporating the default intertwiner boosted overall mean Average Precision (mAP) from 32.8% to 35.2%. Crucially, the accuracy for small objects ($AP_S$) saw the largest jump, increasing by 2.6%.
- Optimal Transport Superiority: Comparing different alignment losses showed that Optimal Transport outperformed standard $L_2$ distance and KL divergence, yielding a +2.0% AP increase over the baseline.
- State-of-the-Art Comparison: On the COCO
test-devset, InterNet achieved 42.5% AP (single-scale) and 44.2% AP (multi-scale), outperforming contemporary models like Mask R-CNN and RetinaNet. - PASCAL VOC: InterNet achieved 73.1% mAP with a VGG-16 backbone and 82.7% mAP with ResNet-101, demonstrating robust improvements over standard Faster R-CNN baselines.
Limitations and Future Work
Despite its performance gains, InterNet introduces some overhead:
- Computational Complexity: Training takes roughly 3.4 days on 8 GPUs, which is slower than standard Mask R-CNN.
- Memory and Inference Overhead: Memory consumption increases from 8.3 GB to 9.6 GB per GPU during training, and inference time increases by about 5% (from 308ms to 325ms per image).
- Context Dependency: As seen with the "couch" example, if the large objects are heavily occluded or visually noisy, the buffer stores inaccurate prototypes, which can degrade performance for that specific class.
For future work, the authors suggest extending the feature intertwiner concept beyond object detection to other domains, such as general data classification, provided a logical division between "reliable" and "less reliable" subsets can be established.
Summary updated