Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection

Feng Liu, Xiaosong Zhang, Zhiliang Peng, Zonghao Guo, Fang Wan, Xiangyang Ji, Qixiang Ye

TL;DR

The imTED (integrally migrated Transformer Encoder-Decoder) framework maximizes the potential of self-supervised pre-training by using both the encoder and the decoder of a Vision Transformer (ViT) for object detection. By replacing randomly initialized components like the Feature Pyramid Network (FPN) and detector heads with pre-trained modules, imTED significantly boosts generalization, achieving state-of-the-art results in few-shot and occluded object detection.

What Problem Does This Paper Address?

In modern computer vision, we almost always use "transfer learning": we take a backbone network (like a ViT or ResNet) pre-trained on a massive dataset (like ImageNet) and "fine-tune" it for a specific task like object detection.

However, there is a catch. While the backbone is pre-trained, other critical parts of the detector—specifically the Feature Pyramid Network (FPN) and the detector head—are typically initialized from scratch with random weights. These randomly initialized components can occupy up to 40% of the model's parameters. This creates a "bottleneck" for generalization; the model has to learn how to use its powerful pre-trained features for localization and classification from a standing start, which is particularly difficult when training data is scarce.

Methods

The authors propose a "fully pre-trained" feature extraction path. Their approach, imTED, leverages models pre-trained with Masked Autoencoders (MAE). In MAE, an encoder learns to represent visible patches, and a decoder learns to reconstruct missing ones.

Key innovations include:

  1. Integral Migration: Instead of discarding the MAE decoder after pre-training, imTED migrates it to serve as the detector head.
  2. Removing the FPN: Standard detectors use an FPN to handle objects at different scales. The authors argue that because Transformers have an adaptive receptive field, a heavy, randomly-initialized FPN is counterproductive. They remove it from the primary feature extraction path to maintain the "fully pre-trained" flow.
  3. Multi-scale Feature Modulator (MFM): To regain scale adaptability without breaking the pre-trained path, they introduce MFM. This module acts after RoI-Align, adaptively weighting single-scale features from the encoder with multi-scale features from a side-branch FPN.
Comparison of baseline vs. imTED architecture

The resulting architecture (shown in Figure 2) ensures that the path from input image to final prediction is as consistent as possible with the original pre-training task.

Figure 2. Architecture of the imTED detector. By integrally migrating the transformer encoder-decoder, imTED constructs a feature extraction path, which is “fully pre-trained”. The reconstructed feature pyramid is only applied for object proposal generation but does not involve the feature extraction procedure. With these designs, the proportion of randomly initialized network parameters of the detector is significantly reduced.

Datasets

The models were evaluated on:

  • MS COCO: The standard benchmark for object detection and instance segmentation (~118k training images).
  • ImageNet Localization Dataset: Used for preliminary experiments to prove that pre-trained decoders help localization.
  • Few-shot COCO: A subset where only 10 or 30 instances per category are available for training.
  • Occluded Object Subset: A custom subset of 534 COCO images containing highly overlapping objects to test robustness.

Key Insights and Ideas

  • The Decoder is a Localization Specialist: The authors found that the MAE decoder, originally designed to reconstruct pixels, is surprisingly good at spatial context modeling. Using a pre-trained decoder as a head improves localization accuracy significantly compared to a randomly initialized one.
  • Consistency Trumps Complexity: Removing the standard FPN (a "multi-scale prior") in favor of a "fully pre-trained" single-scale path actually improved performance. This suggests that maintaining the integrity of pre-trained representations is more important than adding complex, task-specific structures from scratch.
  • Superior Few-Shot Learner: Because imTED starts with almost all its "knowledge" intact, it excels when data is limited. It effectively "remembers" how to handle spatial relationships and occlusions from its self-supervised phase.

Key Results

  • Standard Detection: On MS COCO, imTED outperformed the baseline (a ViT-B backbone with a standard head) by 2.4 AP.
  • Few-Shot Detection: This is where imTED shines. Under a 30-shot setting, it improved the state-of-the-art by a massive 7.6 AP (30.2 vs 22.6).
  • Occlusion Robustness: In tests on occluded objects, imTED improved performance by 2.4 AP. The authors attribute this to the MAE pre-training task, which inherently teaches the model to "fill in the blanks" for missing or hidden parts of an image.
  • Faster Convergence: As seen in Figure 5, imTED reaches higher accuracy much earlier in the training process than standard baselines.
Performance gains and convergence

Limitations and Future Work

  • Computational Overhead: The Transformer decoder is computationally heavier than the simple convolutional layers typically used in detector heads. The authors note a 6.7% increase in FLOPs.
  • Future Directions: The authors suggest exploring knowledge distillation to compress the heavy pre-trained decoder into a lighter-weight version or using cascaded rejection strategies to reduce the number of object proposals the decoder needs to process.

Summary updated

Abstract

Modern object detectors have taken the advantages of backbone networks pre-trained on large scale datasets. Except for the backbone networks, however, other components such as the detector head and the feature pyramid network (FPN) remain trained from scratch, which hinders fully tapping the potential of representation models. In this study, we propose to integrally migrate pre-trained transformer encoder-decoders (imTED) to a detector, constructing a feature extraction path which is ``fully pre-trained" so that detectors' generalization capacity is maximized. The essential differences between imTED with the baseline detector are twofold: (1) migrating the pre-trained transformer decoder to the detector head while removing the randomly initialized FPN from the feature extraction path; and (2) defining a multi-scale feature modulator (MFM) to enhance scale adaptability. Such designs not only reduce randomly initialized parameters significantly but also unify detector training with representation learning intendedly. Experiments on the MS COCO object detection dataset show that imTED consistently outperforms its counterparts by $\sim$2.4 AP. Without bells and whistles, imTED improves the state-of-the-art of few-shot object detection by up to 7.6 AP. Code is available at https://github.com/LiewFeng/imTED.
BibTeX
@inproceedings{liu2023integrally,
  author = {Feng Liu and Xiaosong Zhang and Zhiliang Peng and Zonghao Guo and Fang Wan and Xiangyang Ji and Qixiang Ye},
  title = {{Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00628},
}
Figure