DETR Does Not Need Multi-Scale or Locality Design

Yutong Lin, Yuhui Yuan, Zheng Zhang, Chen Li, Nanning Zheng, Han Hu

Introduction

The original DEtection TRansformer (DETR) was a milestone in computer vision, simplifying object detection by replacing complex, hand-engineered components with a unified transformer architecture. However, to address slow convergence and poor performance on small objects, subsequent state-of-the-art models like Deformable DETR and DINO reintroduced traditional inductive biases, specifically multi-scale feature pyramids and local attention mechanisms. While effective, these additions increased architectural complexity and domain specificity.

This paper investigates whether the original "plain" DETR philosophy—utilizing a single-scale feature map and global cross-attention—can still achieve top-tier performance. The authors demonstrate that by incorporating modern pre-training techniques and a novel spatial bias, architectural simplicity can indeed outperform complex, hand-engineered designs.

Methods: Reclaiming Simplicity

The researchers identify two primary technologies that compensate for the absence of multi-scale features and locality constraints: Box-to-Pixel Relative Position Bias (BoxRPB) and Masked Image Modeling (MIM) pre-training.

Box-to-Pixel Relative Position Bias (BoxRPB)

Standard attention is content-aware but lacks inherent spatial guidance. BoxRPB introduces a spatial bias term to the cross-attention calculation, using the 4D coordinates of a predicted bounding box relative to every pixel in the feature map. To maintain computational efficiency, the authors employ axial decomposition, calculating x-axis and y-axis biases separately before combining them. This reduces the overhead from quadratic to linear complexity relative to the feature map size.

The BoxRPB mechanism uses axial decomposition to efficiently apply spatial bias to global attention.

MIM-based Pre-training

The study finds that Masked Image Modeling (MIM), such as SimMIM, is significantly more effective than standard supervised pre-training for object detection. MIM encourages the backbone to learn fine-grained localization features, which effectively mitigates the model's reliance on high-resolution multi-scale feature maps (e.g., 1/4 or 1/8 scales).

Modernized Baseline

The model also integrates contemporary "best practices" from the DETR ecosystem, including Focal Loss for classification, iterative box refinement, and hybrid matching strategies that combine one-to-one and one-to-many label assignments during training.

Key Results and Insights

The combination of BoxRPB and MIM pre-training allows the plain DETR architecture to achieve state-of-the-art results on the COCO benchmark.

Performance comparison showing the massive gains achieved by improving the Plain DETR architecture compared to Deformable DETR.
  • Massive Performance Gains: Adding BoxRPB improved the plain DETR baseline by +8.9 mAP on COCO, while MIM pre-training added another +7.4 mAP.
  • Competitive SOTA: Utilizing a Swin-L backbone, the "Plain-DETR" reached 63.9 mAP on COCO test-dev, outperforming heavily engineered models like DINO and DETA.
  • Small Object Recovery: Despite the single-scale limitation, the model saw a +2.9 AP improvement for small objects, proving that spatial bias and strong pre-training can replace the need for high-resolution feature pyramids.
  • Global Context: By maintaining global cross-attention, the model avoids windowing artifacts and better captures long-range dependencies compared to local-attention variants.

Conclusion

This work proves that the complex multi-scale and local attention designs prevalent in modern detectors are not strictly necessary. By leveraging the right spatial biases and self-supervised pre-training, a simple, plain DETR architecture can achieve superior performance and better scalability. Future work may explore applying this philosophy to even "plainer" backbones like non-hierarchical ViTs and extending the approach to segmentation tasks.

Summary updated

Abstract

This paper presents an improved DETR detector that maintains a "plain" nature: using a single-scale feature map and global cross-attention calculations without specific locality constraints, in contrast to previous leading DETR-based detectors that reintroduce architectural inductive biases of multi-scale and locality into the decoder. We show that two simple technologies are surprisingly effective within a plain design to compensate for the lack of multi-scale feature maps and locality constraints. The first is a box-to-pixel relative position bias (BoxRPB) term added to the cross-attention formulation, which well guides each query to attend to the corresponding object region while also providing encoding flexibility. The second is masked image modeling (MIM)-based backbone pre-training which helps learn representation with fine-grained localization ability and proves crucial for remedying dependencies on the multi-scale feature maps. By incorporating these technologies and recent advancements in training and problem formation, the improved "plain" DETR showed exceptional improvements over the original DETR detector. By leveraging the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a Swin-L backbone, which is highly competitive with state-of-the-art detectors which all heavily rely on multi-scale feature maps and region-based feature extraction. Code will be available at https://github.com/impiga/Plain-DETR.
BibTeX
@inproceedings{lin2023detr,
  author = {Yutong Lin and Yuhui Yuan and Zheng Zhang and Chen Li and Nanning Zheng and Han Hu},
  title = {{DETR Does Not Need Multi-Scale or Locality Design}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00602},
}
Figure