FP-DETR: Detection Transformer Advanced by Fully Pre-training

Wen Wang, Yang Cao, Jing Zhang, Dacheng TaoICLR 2022 · OpenReview

TL;DR

FP-DETR introduces a "Fully Pre-trained" detection transformer that moves beyond just pre-training the backbone. By using an encoder-only architecture and treating object queries as "visual prompts," the model achieves competitive performance on COCO with significantly fewer parameters and superior robustness to image corruptions.

What Problem Does This Paper Address?

In the world of computer vision, the standard "pre-train on ImageNet, fine-tune on downstream tasks" paradigm has been highly successful. However, modern detection transformers like DETR and Deformable DETR have a "half-baked" implementation of this: they use a pre-trained CNN backbone but train their core 12-layer transformer component from scratch.

This "trained from scratch" approach has two major downsides:

  1. Data Hunger: The model requires massive amounts of data to learn how to use the transformer layers for detection.
  2. Lack of Robustness: Models trained from scratch often struggle with "out-of-distribution" data, such as images with noise, blur, or weather-related corruptions.

The authors identify that the main hurdle is task discrepancy. ImageNet classification uses a single "class token," while object detection requires multiple "object queries" to find and identify various items in a scene.

Methods

The authors propose FP-DETR, which reformulates the training process through three key innovations:

  • Encoder-only Pre-training: Recognizing that transformer decoders are difficult to pre-train on classification tasks (they tend to overfit to the single class token), the authors move to an encoder-only structure for the pre-training phase.
  • Lightweight Multi-scale Tokenizer: Instead of using a heavy, sophisticated CNN backbone like ResNet-50, they replace it with a simple, single-layer convolutional tokenizer. This simplifies the architecture and ensures the transformer encoder does the heavy lifting of feature extraction.
  • Visual Prompting & Task Adaptor: Inspired by the success of "prompting" in NLP (like GPT-3), they treat query positional embeddings as visual prompts. These prompts tell the model where to look. To bridge the gap during fine-tuning, they introduce a Task Adaptor—a lightweight self-attention layer that helps the model understand the relationships between different object queries in a scene.
Pre-training and fine-tuning architecture of FP-DETR

Datasets

The model was evaluated using several standard and specialized benchmarks:

  • Upstream Pre-training: ImageNet-1k and ImageNet-21k.
  • Downstream Detection: COCO 2017 (the gold standard for object detection).
  • Robustness Testing: COCO-C (a version of COCO with 15 types of synthetic corruptions like blur and noise).
  • Small-scale Generalization: Cityscapes (a smaller dataset used to test how well the model learns when data is scarce).

Key Insights and Ideas

  • The Decoder Bottleneck: The paper argues that the transformer decoder is the "weak link" for pre-training. By ditching it in favor of an encoder-only pre-training phase, they can leverage existing vision transformer (ViT) training techniques more effectively.
  • Prompting as a Bridge: One of the most elegant ideas here is viewing object detection as a "prompting" task. If a pre-trained model already knows what objects look like from ImageNet, it just needs a "visual cue" (the prompt) to tell it which part of the image to focus on for a specific object.
  • Efficiency Matters: By simplifying the backbone and fully pre-training the transformer, the authors show that you can get better results with a "Lite" model (11M parameters) than some traditional models with 40M+ parameters.

Key Results

  • Efficiency: FP-DETR-Lite achieves 37.2 AP on COCO with only 11M parameters. For comparison, the original DETR uses 41M parameters to reach 42.0 AP, and Deformable DETR uses 23M parameters for 40.1 AP.
  • Robustness: On the COCO-C benchmark, FP-DETR-Base showed significantly less performance degradation than Deformable DETR, performing best on 14 out of 15 corruption types.
  • Small Data Performance: When trained on the limited Cityscapes dataset, FP-DETR-Lite (11M) matched the performance of Deformable DETR (40M), proving that full pre-training is a massive advantage when data is limited.
Comparison of FP-DETR with state-of-the-art detection transformers on COCO

Limitations and Future Work

The authors acknowledge that while visual prompts are a powerful analogy, they aren't yet as mature as textual prompts in NLP. Currently, FP-DETR still requires a full fine-tuning phase on the downstream task. A promising future direction would be exploring "frozen" pre-trained models where only the visual prompts are tuned, potentially enabling zero-shot or few-shot object detection. Additionally, the authors noted that the interaction between a pre-trained CNN backbone and a randomly initialized transformer is still a complex area that needs more research.

Summary updated

Abstract

Large-scale pre-training has proven to be effective for visual representation learning on downstream tasks, especially for improving robustness and generalization. However, the recently developed detection transformers only employ pre-training on its backbone while leaving the key component, i.e., a 12-layer transformer, being trained from scratch, which prevents the model from above benefits. This separated training paradigm is mainly caused by the discrepancy between the upstream and downstream tasks. To mitigate the issue, we propose FP-DETR, a new method that Fully Pre-Trains an encoder-only transformer and smoothly fine-tunes it for object detection via a task adapter. Inspired by the success of textual prompts in NLP, we treat query positional embeddings as visual prompts to help the model attend to the target area (prompting) and recognize the object. To this end, we propose the task adapter which leverages self-attention to model the contextual relation between object query embedding. Experiments on the challenging COCO dataset demonstrate that our FP-DETR achieves competitive performance. Moreover, it enjoys better robustness to common corruptions and generalization to small-size datasets than state-of-the-art detection transformers. Code will be made publicly available at $\url{https://github.com/encounter1997/FP-DETR}$.
BibTeX
@inproceedings{wang2022fp,
  author = {Wen Wang and Yang Cao and Jing Zhang and Dacheng Tao},
  title = {{FP-DETR: Detection Transformer Advanced by Fully Pre-training}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=JgQ9SjHh60u},
}
Figure