TL;DR
This paper introduces Self-EMD, a novel self-supervised learning method for object detection that trains directly on non-iconic datasets like COCO, bypassing the need for ImageNet. By preserving spatial feature maps and using Earth Mover's Distance (EMD) for similarity, Self-EMD achieves competitive performance with ImageNet-pretrained models, demonstrating the potential of readily available unlabeled data.
What Problem Does This Paper Address?
Traditional self-supervised learning (SSL) for computer vision, particularly for object detection, heavily relies on pre-training on the ImageNet dataset. ImageNet is an "iconic-object" dataset, meaning images are typically centered around a single object. This characteristic is crucial for SSL methods that treat different "views" or crops of an image as representing the same object. However, collecting and cleaning such large datasets is costly.
The problem arises when applying these SSL methods to "non-iconic" datasets like COCO, which are easier to obtain but contain multiple objects per image. In COCO, random crops might capture entirely different objects, breaking the core assumption of many SSL approaches and leading to degraded performance. Furthermore, many SSL methods use global pooling to generate feature embeddings, which discards crucial spatial information needed for object detection. This paper aims to bridge this gap by developing an SSL method that works effectively on non-iconic datasets and preserves spatial features for better object detection.
Methods
The core innovation of Self-EMD lies in two key aspects:
- Preserving Spatial Feature Maps: Instead of using global pooling layers (common in methods like BYOL) that condense features into a single vector, Self-EMD retains the full convolutional feature maps. This preserves the spatial structure and local information, which is vital for object detection.
- Earth Mover's Distance (EMD) for Similarity: To measure the similarity between these spatial feature maps from different image crops, Self-EMD employs the Earth Mover's Distance (EMD). EMD is a metric that calculates the minimum "cost" to transform one distribution into another. In this context, it finds the optimal matching between local feature patches across two crops, even if they contain different objects at different locations. This allows the model to learn meaningful correspondences without explicit labels.
The method builds upon the BYOL framework but modifies its output and similarity metric. It also introduces:
- Scale Invariant Training: To address the challenge of varying object scales in detection, Self-EMD uses an additional downscaled view of the image and incorporates Spatial Pyramid Cropping (SPC) to compare features across different scales.
- Marginal Weights: For EMD, specific marginal weights are designed to control the importance of different feature locations, adapting to the unsupervised setting.
Datasets
- Training: The primary training dataset used is the COCO 2017 train set (unlabeled). The paper also experiments with an augmented dataset called "COCO+", which combines the COCO 2017 train set with an additional unlabeled COCO set. For comparison and ablation studies, ImageNet (labeled) is also used as a pre-training dataset.
- Evaluation: Performance is evaluated on the COCO val 2017 and Pascal VOC 2007 test datasets.
Key Insights and Ideas
- ImageNet Bias is a Bottleneck: The paper strongly argues that the success of many current SSL methods is heavily tied to the "object-centric" bias of ImageNet. When this bias is absent (as in COCO), performance suffers significantly, highlighting a limitation of existing SSL paradigms for real-world, diverse datasets.
- Spatial Information is King for Detection: Global pooling, while useful for general representation learning, actively harms object detection performance by discarding spatial context. Retaining feature maps is crucial for downstream detection tasks.
- EMD as a Robust Similarity Metric: EMD proves to be a powerful tool for comparing feature maps from non-iconic images. Its ability to find optimal local correspondences makes it robust to variations in object presence and location, a key advantage over simple L2 distance.
- Unsupervised Learning Can Compete with Supervised: Self-EMD demonstrates that with the right approach, unsupervised pre-training on readily available, unlabeled data can achieve performance comparable to, and sometimes exceeding, supervised pre-training on large, labeled datasets like ImageNet for object detection.
Key Results
- Competitive Performance on COCO: Using a Faster R-CNN (ResNet50-FPN) detector, Self-EMD pre-trained on the unlabeled COCO train set achieves 39.8% mAP, which is on par with state-of-the-art SSL methods pre-trained on ImageNet. This surpasses BYOL pre-trained on COCO (38.8% mAP).
- Improved Performance with More Data: Further pre-training on the "COCO+" dataset (241K images) boosts Self-EMD's performance to 40.4% mAP, showcasing the benefits of leveraging more unlabeled data.
- Outperforms BYOL on Non-Iconic Data: When pre-trained on ImageNet, Self-EMD and BYOL perform similarly. However, when pre-trained on COCO, BYOL's performance drops significantly, while Self-EMD maintains its strong performance.
- Benefits Various Detectors: Self-EMD shows improvements across different detectors like Mask R-CNN and RetinaNet, with particularly significant gains for single-stage detectors like RetinaNet, likely due to their reliance on dense local features.
- Pascal VOC Performance: On Pascal VOC, Self-EMD outperforms BYOL by 3.0% on AP and 4.2% on AP50.
Limitations and Future Work
- Gap with Supervised Pre-training: Despite achieving competitive results, the paper acknowledges a significant gap between all current self-supervised methods and supervised pre-training on ImageNet for object detection. This suggests that while progress is being made, there's still room for improvement in the self-supervised domain.
- Scale Invariant Training Improvement: The improvement from scale invariant training (SPC and multi-scale training) was modest (0.3% AP). The authors speculate that existing data augmentation in SSL might already introduce some scale invariance, and future work could focus on designing more explicit scale-invariant data augmentation strategies.
- Generalizability to Other Backbones: While Self-EMD shows promise with different backbones (MobileNetV2, ResNet101), its performance relative to supervised pre-training is still inferior for smaller models like MobileNetV2.
The paper concludes by highlighting the potential of Self-EMD for leveraging more unlabeled data and suggests that further exploration in self-supervised training frameworks is needed to close the gap with supervised methods.
Summary updated