Recurrent Glimpse-based Decoder for Detection with Transformer

Zhe Chen, Jing Zhang, Dacheng Tao

Introduction

Detection Transformers (DETR) have fundamentally changed object detection by eliminating the need for hand-crafted components like Non-Maximum Suppression (NMS) and anchor generation. However, they suffer from a significant drawback: slow training convergence. Because Transformers rely on global attention, they must learn to identify objects from scratch across the entire image, lacking the "locality inductive bias" inherent in convolutional neural networks. This often results in models requiring hundreds of GPU hours to achieve competitive accuracy.

To address this, we introduce the Recurrent Glimpse-based decOder (REGO). REGO is a plug-and-play module designed to accelerate DETR training by implementing a multi-stage, "coarse-to-fine" refinement process. By focusing on local "glimpses" of an image, REGO provides the necessary locality bias to guide the model's attention more efficiently.

Figure 1: Overview of the REGO framework showing how the multi-stage glimpse process refines initial coarse detections into robust results.

Methodology

REGO introduces a recurrent processing structure that can be integrated into various DETR architectures. The workflow mimics a human-like "glimpse" strategy:

  1. Multi-stage Recurrence: The model processes detections over three sequential stages rather than a single pass.
  2. Glimpse Extraction: In each stage, the model utilizes bounding boxes predicted in the previous stage. These boxes are slightly enlarged to include surrounding context, and visual features are extracted specifically from these Regions of Interest (RoIs).
  3. Glimpse-based Decoder: A dedicated decoder uses cross-attention to correlate these local glimpse features with the attention outputs from the preceding stage.
  4. Coarse-to-Fine Refinement: Early stages use larger glimpses to capture context and correct initial errors, while later stages narrow the focus to refine precise object boundaries.
  5. Residual Updates: Instead of predicting entirely new coordinates, the model predicts refinements (offsets) to the previous stage's boxes, ensuring spatial consistency.

Experimental Setup

The effectiveness of REGO was evaluated using the MS COCO dataset (118k training, 5k validation images). Testing spanned multiple backbones, including ResNet-50, ResNet-101, and ResNeXt-101, to demonstrate the module's versatility across different feature extraction scales.

Key Results and Discussion

REGO significantly outperforms baseline DETR models in both training speed and final accuracy. Notably, REGO enabled Deformable DETR to reach 44.8 AP (Average Precision) in just 36 epochs. In comparison, the original DETR required 500 epochs to reach a lower score of 42.0 AP, representing a 13x reduction in training time.

Figure 4: Comparison of training convergence between Deformable DETR and REGO-augmented Deformable DETR, highlighting the rapid AP gains.

Beyond speed, REGO provides a substantial performance boost. When constrained to a standard 50-epoch training limit, REGO improved various DETR detectors by up to a 7% relative gain. Using a ResNeXt-101 backbone, the REGO-augmented model achieved 49.1 AP, surpassing established detectors like Mask R-CNN and Cascade Mask R-CNN.

Qualitative analysis suggests that the glimpse decoder learns semantic relationships; for instance, it learns to correlate a "fork" glimpse with a nearby "pizza" glimpse. Furthermore, models trained with REGO retain improved performance even if the module is removed during inference, indicating that REGO helps the base Transformer learn superior global attention patterns during the training phase.

Conclusion

The Recurrent Glimpse-based Decoder (REGO) effectively mitigates the slow convergence of DETR by introducing a locality-focused refinement strategy. By adding only 10% more computational complexity, REGO achieves state-of-the-art results with a fraction of the training time. Future work may explore applying this recurrent refinement to related tasks such as instance segmentation and video object detection.

Reference: Chen, Z., Zhang, J., & Tao, D. (2022). Recurrent Glimpse-based Decoder for Detection with Transformer. arXiv:2112.04632v2.

Summary updated

Abstract

Although detection with Transformer (DETR) is increasingly popular, its global attention modeling requires an extremely long training period to optimize and achieve promising detection performance. Alternative to existing studies that mainly develop advanced feature or embedding designs to tackle the training issue, we point out that the Region-of-Interest (RoI) based detection refinement can easily help mitigate the difficulty of training for DETR methods. Based on this, we introduce a novel REcurrent Glimpse-based decOder (REGO) in this paper. In particular, the REGO employs a multi-stage recurrent processing structure to help the attention of DETR gradually focus on foreground objects more accurately. In each processing stage, visual features are extracted as glimpse features from RoIs with enlarged bounding box areas of detection results from the previous stage. Then, a glimpse-based decoder is introduced to provide refined detection results based on both the glimpse features and the attention modeling outputs of the previous stage. In practice, REGO can be easily embedded in representative DETR variants while maintaining their fully end-to-end training and inference pipelines. In particular, REGO helps Deformable DETR achieve 44.8 AP on the MSCOCO dataset with only 36 training epochs, compared with the first DETR and the Deformable DETR that require 500 and 50 epochs to achieve comparable performance, respectively. Experiments also show that REGO consistently boosts the performance of different DETR detectors by up to 7% relative gain at the same setting of 50 training epochs. Code is available via https://github.com/zhechen/Deformable-DETR-REGO.
BibTeX
@inproceedings{chen2021recurrent,
  author = {Zhe Chen and Jing Zhang and Dacheng Tao},
  title = {{Recurrent Glimpse-based Decoder for Detection with Transformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr52688.2022.00519},
}
Figure