Introduction
The original DEtection TRansformer (DETR) was a milestone in computer vision, simplifying object detection by replacing complex, hand-engineered components with a unified transformer architecture. However, to address slow convergence and poor performance on small objects, subsequent state-of-the-art models like Deformable DETR and DINO reintroduced traditional inductive biases, specifically multi-scale feature pyramids and local attention mechanisms. While effective, these additions increased architectural complexity and domain specificity.
This paper investigates whether the original "plain" DETR philosophy—utilizing a single-scale feature map and global cross-attention—can still achieve top-tier performance. The authors demonstrate that by incorporating modern pre-training techniques and a novel spatial bias, architectural simplicity can indeed outperform complex, hand-engineered designs.
Methods: Reclaiming Simplicity
The researchers identify two primary technologies that compensate for the absence of multi-scale features and locality constraints: Box-to-Pixel Relative Position Bias (BoxRPB) and Masked Image Modeling (MIM) pre-training.
Box-to-Pixel Relative Position Bias (BoxRPB)
Standard attention is content-aware but lacks inherent spatial guidance. BoxRPB introduces a spatial bias term to the cross-attention calculation, using the 4D coordinates of a predicted bounding box relative to every pixel in the feature map. To maintain computational efficiency, the authors employ axial decomposition, calculating x-axis and y-axis biases separately before combining them. This reduces the overhead from quadratic to linear complexity relative to the feature map size.
MIM-based Pre-training
The study finds that Masked Image Modeling (MIM), such as SimMIM, is significantly more effective than standard supervised pre-training for object detection. MIM encourages the backbone to learn fine-grained localization features, which effectively mitigates the model's reliance on high-resolution multi-scale feature maps (e.g., 1/4 or 1/8 scales).
Modernized Baseline
The model also integrates contemporary "best practices" from the DETR ecosystem, including Focal Loss for classification, iterative box refinement, and hybrid matching strategies that combine one-to-one and one-to-many label assignments during training.
Key Results and Insights
The combination of BoxRPB and MIM pre-training allows the plain DETR architecture to achieve state-of-the-art results on the COCO benchmark.
- Massive Performance Gains: Adding BoxRPB improved the plain DETR baseline by +8.9 mAP on COCO, while MIM pre-training added another +7.4 mAP.
- Competitive SOTA: Utilizing a Swin-L backbone, the "Plain-DETR" reached 63.9 mAP on COCO test-dev, outperforming heavily engineered models like DINO and DETA.
- Small Object Recovery: Despite the single-scale limitation, the model saw a +2.9 AP improvement for small objects, proving that spatial bias and strong pre-training can replace the need for high-resolution feature pyramids.
- Global Context: By maintaining global cross-attention, the model avoids windowing artifacts and better captures long-range dependencies compared to local-attention variants.
Conclusion
This work proves that the complex multi-scale and local attention designs prevalent in modern detectors are not strictly necessary. By leveraging the right spatial biases and self-supervised pre-training, a simple, plain DETR architecture can achieve superior performance and better scalability. Future work may explore applying this philosophy to even "plainer" backbones like non-hierarchical ViTs and extending the approach to segmentation tasks.
Summary updated