Introduction
Detection Transformers (DETR) have fundamentally changed object detection by eliminating the need for hand-crafted components like Non-Maximum Suppression (NMS) and anchor generation. However, they suffer from a significant drawback: slow training convergence. Because Transformers rely on global attention, they must learn to identify objects from scratch across the entire image, lacking the "locality inductive bias" inherent in convolutional neural networks. This often results in models requiring hundreds of GPU hours to achieve competitive accuracy.
To address this, we introduce the Recurrent Glimpse-based decOder (REGO). REGO is a plug-and-play module designed to accelerate DETR training by implementing a multi-stage, "coarse-to-fine" refinement process. By focusing on local "glimpses" of an image, REGO provides the necessary locality bias to guide the model's attention more efficiently.
Methodology
REGO introduces a recurrent processing structure that can be integrated into various DETR architectures. The workflow mimics a human-like "glimpse" strategy:
- Multi-stage Recurrence: The model processes detections over three sequential stages rather than a single pass.
- Glimpse Extraction: In each stage, the model utilizes bounding boxes predicted in the previous stage. These boxes are slightly enlarged to include surrounding context, and visual features are extracted specifically from these Regions of Interest (RoIs).
- Glimpse-based Decoder: A dedicated decoder uses cross-attention to correlate these local glimpse features with the attention outputs from the preceding stage.
- Coarse-to-Fine Refinement: Early stages use larger glimpses to capture context and correct initial errors, while later stages narrow the focus to refine precise object boundaries.
- Residual Updates: Instead of predicting entirely new coordinates, the model predicts refinements (offsets) to the previous stage's boxes, ensuring spatial consistency.
Experimental Setup
The effectiveness of REGO was evaluated using the MS COCO dataset (118k training, 5k validation images). Testing spanned multiple backbones, including ResNet-50, ResNet-101, and ResNeXt-101, to demonstrate the module's versatility across different feature extraction scales.
Key Results and Discussion
REGO significantly outperforms baseline DETR models in both training speed and final accuracy. Notably, REGO enabled Deformable DETR to reach 44.8 AP (Average Precision) in just 36 epochs. In comparison, the original DETR required 500 epochs to reach a lower score of 42.0 AP, representing a 13x reduction in training time.
Beyond speed, REGO provides a substantial performance boost. When constrained to a standard 50-epoch training limit, REGO improved various DETR detectors by up to a 7% relative gain. Using a ResNeXt-101 backbone, the REGO-augmented model achieved 49.1 AP, surpassing established detectors like Mask R-CNN and Cascade Mask R-CNN.
Qualitative analysis suggests that the glimpse decoder learns semantic relationships; for instance, it learns to correlate a "fork" glimpse with a nearby "pizza" glimpse. Furthermore, models trained with REGO retain improved performance even if the module is removed during inference, indicating that REGO helps the base Transformer learn superior global attention patterns during the training phase.
Conclusion
The Recurrent Glimpse-based Decoder (REGO) effectively mitigates the slow convergence of DETR by introducing a locality-focused refinement strategy. By adding only 10% more computational complexity, REGO achieves state-of-the-art results with a fraction of the training time. Future work may explore applying this recurrent refinement to related tasks such as instance segmentation and video object detection.
Reference: Chen, Z., Zhang, J., & Tao, D. (2022). Recurrent Glimpse-based Decoder for Detection with Transformer. arXiv:2112.04632v2.
Summary updated