Integral Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection

Xiaosong Zhang, Feng Liu, Zhiliang Peng, Zonghao Guo, Fang Wan, Xiangyang Ji, Qixiang Ye

TL;DR

This paper proposes "Integrally Migrating Pre-trained Transformer Encoder-decoders" (imTED) for object detection. By migrating both the pre-trained encoder and decoder of vision transformers, imTED creates a "fully pre-trained" feature extraction path, significantly boosting generalization capabilities and outperforming existing methods, especially in low-shot scenarios.

What Problem Does This Paper Address?

Modern object detectors, particularly those leveraging powerful pre-trained vision transformers (like ViTs), typically only utilize the transformer's encoder as a backbone. The subsequent components, such as the Feature Pyramid Network (FPN) and the detector head, are still trained from scratch. This creates a disconnect: the backbone is highly optimized and generalized from massive datasets, while the rest of the pipeline is initialized randomly and trained on smaller, specific datasets. This inconsistency limits the full potential of pre-trained models, hindering the detector's overall generalization capability. The authors aim to bridge this gap by creating a more cohesive and "fully pre-trained" feature extraction pipeline.

Methods

The core innovation is the Integrally Migrating Pre-trained Transformer Encoder-Decoders (imTED). Instead of just using the pre-trained ViT encoder, imTED also incorporates the pre-trained ViT decoder as the detector head. This means the entire path from input to object prediction is built upon pre-trained components.

Key methodological changes include:

  • Migrating the Pre-trained Decoder: The MAE-pre-trained ViT decoder is repurposed as the detector head, replacing the randomly initialized components typically found there. This allows the decoder's inherent ability to model spatial context and relationships to directly benefit object localization.
  • Removing the FPN: The traditional Feature Pyramid Network (FPN), often used to create multi-scale features, is removed from the feature extraction path. The authors argue that the inherent multi-scale receptive field capabilities of the transformer encoder, combined with the decoder's context modeling, make FPN redundant for feature extraction.
  • Multi-Scale Feature Modulator (MFM): To reintroduce multi-scale adaptivity without a full FPN, a novel MFM is proposed. This module adaptively combines single-scale features (from the imTED backbone) with multi-scale features (generated by a lightweight, randomly initialized FPN) through a learnable weighting mechanism. This allows for scale adaptation without compromising the "fully pre-trained" nature of the core feature extraction.

Datasets

  • ImageNet-1K: Used for pre-training the ViT models (encoder and decoder) using the Masked Autoencoder (MAE) method.
  • MS COCO: The primary dataset for evaluating object detection performance. This includes standard training and validation splits, as well as subsets for low-shot and occluded object detection experiments.
  • ImageNet Localization Dataset: Used for initial motivation experiments to demonstrate the benefit of a pre-trained decoder for object localization.

Key Insights and Ideas

  • The Decoder is Key for Localization: The paper strongly suggests that the decoder, often discarded in prior work, plays a crucial role in object localization by leveraging its ability to model image context. This is a significant departure from previous approaches that focused solely on the encoder for feature extraction.
  • "Fully Pre-trained" Path is More Important than FPN: Removing the FPN, a standard component for multi-scale detection, and relying on the integral migration of the encoder-decoder leads to significant performance gains. This highlights the paramount importance of consistent, pre-trained feature extraction over architectural components trained from scratch.
  • Transformer's Intrinsic Adaptability: The inherent adaptive receptive field of transformers reduces the reliance on explicit multi-scale feature fusion mechanisms like FPN, simplifying the architecture while maintaining performance.
  • Unifying Representation Learning and Detection: imTED moves towards a more unified approach where the representation learning (pre-training) directly extends into the detection pipeline, reducing the gap between these two stages.

Key Results

  • Improved Generalization: imTED consistently outperforms baseline detectors (e.g., Faster R-CNN with ViT-B backbone) by approximately 2.4% AP on the MS COCO dataset.
  • State-of-the-Art Few-Shot Detection: imTED achieves significant improvements in few-shot object detection, reaching up to 7.6% AP gains over existing methods.
  • Reduced Randomly Initialized Parameters: The imTED approach drastically reduces the number of randomly initialized parameters, with an example showing an 81.3% reduction compared to a standard Faster R-CNN with a ViT backbone.
  • Effective Occlusion Handling: The method shows strong performance on occluded object detection, benefiting from the MAE's inherent ability to handle masked (occluded) patches during pre-training.
  • Faster Convergence: The imTED detector shows faster convergence during training, especially for localization loss.

Limitations and Future Work

  • Computational Cost of Decoder: While the overall architecture is simplified, the computational cost of the decoder can be moderately larger than a traditional Conv-based detector head.
  • Future Directions:
    • Cascaded Rejection Strategies: To further optimize performance, the authors suggest exploring cascaded rejection strategies for object proposals.
  • Light-weight Decoders: Investigating knowledge distillation to create lighter-weight decoders is proposed as another avenue for efficiency.

Summary updated

Abstract

Modern object detectors have taken the advantages of pre-trained vision transformers by using them as backbone networks. However, except for the backbone networks, other detector components, such as the detector head and the feature pyramid network, remain randomly initialized, which hinders the consistency between detectors and pre-trained models. In this study, we propose to integrally migrate the pre-trained transformer encoder-decoders (imTED) for object detection, constructing a feature extraction-operation path that is not only "fully pre-trained" but also consistent with pre-trained models. The essential improvements of imTED over existing transformer-based detectors are twofold: (1) it embeds the pre-trained transformer decoder to the detector head; and (2) it removes the feature pyramid network from the feature extraction path. Such improvements significantly reduce the proportion of randomly initialized parameters and enhance the generation capability of detectors. Experiments on MS COCO dataset demonstrate that imTED consistently outperforms its counterparts by ~2.8% AP. Without bells and whistles, imTED improves the state-of-the-art of few-shot object detection by up to 7.6% AP, demonstrating significantly higher generalization capability. Code will be made publicly available.
BibTeX
@inproceedings{zhang2023integral,
  author = {Xiaosong Zhang and Feng Liu and Zhiliang Peng and Zonghao Guo and Fang Wan and Xiangyang Ji and Qixiang Ye},
  title = {{Integral Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00628},
}
Figure