Abstract
In this paper, we present a novel Dynamic DETR (Detection with Transformers) approach by introducing dynamic attentions into both the encoder and decoder stages of DETR to break its two limitations on small feature resolution and slow training convergence. To address the first limitation, which is due to the quadratic computational complexity of the self-attention module in Transformer encoders, we propose a dynamic encoder to approximate the Transformer encoder’s attention mechanism using a convolution-based dynamic encoder with various attention types. Such an encoder can dynamically adjust attentions based on multiple factors such as scale importance, spatial importance, and representation (i.e., feature dimension) importance. To mitigate the second limitation of learning difficulty, we introduce a dynamic decoder by replacing the cross-attention module with a ROI-based dynamic attention in the Transformer decoder. Such a decoder effectively assists Transformers to focus on region of interests from a coarse-to-fine manner and dramatically lowers the learning difficulty, leading to a much faster convergence with fewer training epochs. We conduct a series of experiments to demonstrate our advantages. Our Dynamic DETR significantly reduces the training epochs (by 14×), yet results in a much better performance (by 3.6 on mAP). Meanwhile, in the standard 1× setup with ResNet-50 backbone, we archive a new state-of-the-art performance that further proves the learning effectiveness of the proposed approach.
Introduction
The original DEtection TRansformer (DETR) was a breakthrough in object detection, treating the task as a direct set-prediction problem and eliminating the need for hand-crafted components like Non-Maximum Suppression (NMS) or anchor boxes. However, DETR faced two significant challenges: high computational complexity due to the quadratic scaling ($O(N^2)$) of self-attention, which limited its ability to process high-resolution features, and slow training convergence, often requiring 500 epochs to reach peak performance.
Dynamic DETR addresses these issues by introducing dynamic attention mechanisms in both the encoder and decoder stages. This approach allows for efficient processing of multi-scale feature pyramids and drastically accelerates the learning process by focusing on relevant image regions.
Methodology
Dynamic DETR introduces two primary architectural innovations to overcome the limitations of the standard Transformer architecture in detection tasks.
Dynamic Encoder
The Dynamic Encoder replaces standard Transformer self-attention with a more efficient convolution-based approach. It utilizes Deformable Convolutions to capture spatial importance and Squeeze-and-Excitation (SE) blocks to weight different scales within the feature pyramid. Additionally, Dynamic ReLU is employed to adjust feature representations per-channel. This design allows the model to process full-scale feature pyramids with linear complexity relative to the number of pixels, facilitating the detection of small objects.
Dynamic Decoder
In the original DETR, object queries attend to the entire feature map via cross-attention, which is difficult to learn. Dynamic DETR replaces this with ROI-based dynamic attention. Using current bounding box predictions, the decoder "crops" local features via ROI Pooling. It then generates dynamic filters from the object query to convolve with these local features. This creates a coarse-to-fine refinement process that significantly reduces the search space and accelerates convergence.
Experimental Results
The model was rigorously evaluated on the MS COCO dataset (train2017, val2017, and test-dev). The results demonstrate that Dynamic DETR is not only more accurate but also significantly more efficient than its predecessors.
Convergence and Accuracy
Dynamic DETR achieves a state-of-the-art 47.2 mAP in just 40 epochs, representing a 14x speedup in convergence compared to the original DETR. Even compared to Deformable DETR, which requires 150 epochs to reach similar performance, Dynamic DETR shows superior efficiency. In a standard "1x" training schedule (12 epochs), Dynamic DETR with a ResNet-50 backbone achieved 42.9 mAP, outperforming both Faster R-CNN (37.9 mAP) and the original DETR (15.5 mAP) in the same timeframe.
Scalability
When utilizing a ResNeXt-101-DCN backbone, the model reaches 49.3 mAP on the COCO test-dev set, establishing a new competitive benchmark for end-to-end detectors.
Conclusion
Dynamic DETR successfully mitigates the primary drawbacks of the Transformer-based detection paradigm. By replacing global attention with targeted dynamic mechanisms, the model achieves faster convergence and better performance on multi-scale objects. Future work may involve extending this dynamic framework to related tasks such as instance segmentation and video object detection, as well as optimizing the balance between architectural depth and real-time inference speed.
Summary updated