TL;DR
This paper introduces 2DRCL, a novel pre-training framework for object detection that tackles the challenges of long-tailed data distributions and simplicity bias. By combining holistic and local contrastive learning with dynamic rebalancing and dual reconstruction, 2DRCL significantly improves performance on rare object classes without sacrificing accuracy on more frequent ones.
What Problem Does This Paper Address?
Object detection models often struggle with "long-tailed" datasets, where a few classes have many examples (head classes) while most classes have very few (tail classes). Traditional pre-training methods, which typically rely on uniform or random sampling, fail to adequately represent these tail classes. This imbalance, coupled with "simplicity bias" (where models favor simpler patterns and ignore complex features crucial for tail classes), leads to poor detection performance on rare objects. Existing methods for handling long-tailed data often focus on fine-tuning or specialized loss functions, but this paper aims to address the problem at the pre-training stage, making it more fundamental and broadly applicable.
Methods
The core of the proposed method is the Dynamic Rebalancing Contrastive Learning with Dual Reconstruction (2DRCL) framework. It comprises three main components:
- Holistic-Local Contrastive Learning (HLCL): This component bridges the gap between general visual representation learning and object detection specifics.
- Holistic Contrastive Learning (HCL): Learns general visual features from image-level representations, similar to MoCo.
- Local Contrastive Learning (LCL): Focuses on object-level features by using object proposals (bounding boxes) generated by a class-agnostic detector. This aligns pre-training more closely with the downstream detection task.
- Dynamic Rebalancing: This strategy addresses the data imbalance. Instead of simple resampling, it dynamically adjusts sampling probabilities at both the image and object proposal levels throughout pre-training. The focus shifts from image-level balancing to instance-level balancing as training progresses, giving more attention to tail classes.
- Dual Reconstruction (DRC): This component tackles simplicity bias.
- Appearance Reconstruction (AR): Enforces pixel-level reconstruction of input images to ensure the model captures fine-grained visual details.
- Semantic Reconstruction (SR): Uses masked input images to ensure semantic consistency between original and reconstructed features, forcing the model to learn deeper semantic relationships beyond superficial appearances.
These components are integrated into an end-to-end pre-training framework.
Datasets
- COCO [33]: Used for evaluating performance under balanced class distributions. It contains 80 classes.
- LVIS v1.0 [11]: Used for evaluating performance under long-tailed class distributions. It features 1,203 classes categorized into rare, common, and frequent based on instance counts.
Key Insights and Ideas
- Pre-training matters for long-tailed detection: The paper argues that addressing long-tailed issues at the pre-training stage is crucial for robust object detection, rather than solely relying on fine-tuning.
- Holistic + Local is better: Combining general image-level features (HCL) with object-specific features (LCL) better aligns pre-training with the demands of object detection.
- Dynamic rebalancing is key: A dynamic, instance-aware rebalancing strategy is more effective than static resampling for tail classes.
- Dual Reconstruction combats simplicity bias: Simultaneously reconstructing appearance (AR) and semantics (SR) forces the model to learn richer, more robust features essential for distinguishing rare objects.
- Trade-off between accuracy and computational cost: While 2DRCL offers significant performance gains, it comes with increased VRAM usage and training time compared to some baselines.
Key Results
- COCO: 2DRCL achieved the highest APbb (41.4%) and APmk (37.3%) on COCO, outperforming methods like AlignDet and supervised baselines, even with fewer pre-training epochs.
- LVIS v1.0:
- Compared to other self-supervised pre-training methods (MoCo v2, BYOL, etc.), 2DRCL showed significantly better APbb scores, especially for tail classes.
- When combined with existing long-tailed fine-tuning methods, 2DRCL consistently improved performance, achieving state-of-the-art results on LVIS v1.0. For example, it surpassed competitors by large margins in APbb and AP.
- Error Analysis: 2DRCL showed noticeable improvements for rare object classes, reducing non-background and background classification errors, while only slightly impacting accuracy on frequent classes.
- Simplicity Bias Analysis: Visualizations of attention maps demonstrated that 2DRCL effectively mitigates simplicity bias, capturing more informative regions for tail classes and reducing background interference.
Limitations and Future Work
- Computational Cost: The authors acknowledge that 2DRCL has relatively high computational costs (VRAM and training time) compared to some methods.
- Future Work: The authors plan to focus on optimizing computational efficiency in future work.
Summary updated