TL;DR
This paper proposes "Integrally Migrating Pre-trained Transformer Encoder-decoders" (imTED) for object detection. By migrating both the pre-trained encoder and decoder of vision transformers, imTED creates a "fully pre-trained" feature extraction path, significantly boosting generalization capabilities and outperforming existing methods, especially in low-shot scenarios.
What Problem Does This Paper Address?
Modern object detectors, particularly those leveraging powerful pre-trained vision transformers (like ViTs), typically only utilize the transformer's encoder as a backbone. The subsequent components, such as the Feature Pyramid Network (FPN) and the detector head, are still trained from scratch. This creates a disconnect: the backbone is highly optimized and generalized from massive datasets, while the rest of the pipeline is initialized randomly and trained on smaller, specific datasets. This inconsistency limits the full potential of pre-trained models, hindering the detector's overall generalization capability. The authors aim to bridge this gap by creating a more cohesive and "fully pre-trained" feature extraction pipeline.
Methods
The core innovation is the Integrally Migrating Pre-trained Transformer Encoder-Decoders (imTED). Instead of just using the pre-trained ViT encoder, imTED also incorporates the pre-trained ViT decoder as the detector head. This means the entire path from input to object prediction is built upon pre-trained components.
Key methodological changes include:
- Migrating the Pre-trained Decoder: The MAE-pre-trained ViT decoder is repurposed as the detector head, replacing the randomly initialized components typically found there. This allows the decoder's inherent ability to model spatial context and relationships to directly benefit object localization.
- Removing the FPN: The traditional Feature Pyramid Network (FPN), often used to create multi-scale features, is removed from the feature extraction path. The authors argue that the inherent multi-scale receptive field capabilities of the transformer encoder, combined with the decoder's context modeling, make FPN redundant for feature extraction.
- Multi-Scale Feature Modulator (MFM): To reintroduce multi-scale adaptivity without a full FPN, a novel MFM is proposed. This module adaptively combines single-scale features (from the imTED backbone) with multi-scale features (generated by a lightweight, randomly initialized FPN) through a learnable weighting mechanism. This allows for scale adaptation without compromising the "fully pre-trained" nature of the core feature extraction.
Datasets
- ImageNet-1K: Used for pre-training the ViT models (encoder and decoder) using the Masked Autoencoder (MAE) method.
- MS COCO: The primary dataset for evaluating object detection performance. This includes standard training and validation splits, as well as subsets for low-shot and occluded object detection experiments.
- ImageNet Localization Dataset: Used for initial motivation experiments to demonstrate the benefit of a pre-trained decoder for object localization.
Key Insights and Ideas
- The Decoder is Key for Localization: The paper strongly suggests that the decoder, often discarded in prior work, plays a crucial role in object localization by leveraging its ability to model image context. This is a significant departure from previous approaches that focused solely on the encoder for feature extraction.
- "Fully Pre-trained" Path is More Important than FPN: Removing the FPN, a standard component for multi-scale detection, and relying on the integral migration of the encoder-decoder leads to significant performance gains. This highlights the paramount importance of consistent, pre-trained feature extraction over architectural components trained from scratch.
- Transformer's Intrinsic Adaptability: The inherent adaptive receptive field of transformers reduces the reliance on explicit multi-scale feature fusion mechanisms like FPN, simplifying the architecture while maintaining performance.
- Unifying Representation Learning and Detection: imTED moves towards a more unified approach where the representation learning (pre-training) directly extends into the detection pipeline, reducing the gap between these two stages.
Key Results
- Improved Generalization: imTED consistently outperforms baseline detectors (e.g., Faster R-CNN with ViT-B backbone) by approximately 2.4% AP on the MS COCO dataset.
- State-of-the-Art Few-Shot Detection: imTED achieves significant improvements in few-shot object detection, reaching up to 7.6% AP gains over existing methods.
- Reduced Randomly Initialized Parameters: The imTED approach drastically reduces the number of randomly initialized parameters, with an example showing an 81.3% reduction compared to a standard Faster R-CNN with a ViT backbone.
- Effective Occlusion Handling: The method shows strong performance on occluded object detection, benefiting from the MAE's inherent ability to handle masked (occluded) patches during pre-training.
- Faster Convergence: The imTED detector shows faster convergence during training, especially for localization loss.
Limitations and Future Work
- Computational Cost of Decoder: While the overall architecture is simplified, the computational cost of the decoder can be moderately larger than a traditional Conv-based detector head.
- Future Directions:
- Cascaded Rejection Strategies: To further optimize performance, the authors suggest exploring cascaded rejection strategies for object proposals.
- Light-weight Decoders: Investigating knowledge distillation to create lighter-weight decoders is proposed as another avenue for efficiency.
Summary updated