Introduction
The DEtection TRansformer (DETR) revolutionized object detection by eliminating hand-crafted components like Non-Maximum Suppression (NMS) and anchor boxes, framing detection as a direct set-prediction problem. Despite its elegance, DETR faced two significant hurdles: poor performance on small objects and exceptionally slow training convergence, often requiring 500 epochs to reach competitive accuracy.
CF-DETR addresses these challenges by introducing a coarse-to-fine refinement strategy. The authors observe that DETR's struggle is not in "seeing" objects, but in accurately localizing them. By integrating local multi-scale features into the global transformer decoder, the model achieves state-of-the-art performance (47.8 AP on COCO with a ResNet-50) in just 36 epochs—a 14x speedup over the original model.
Methodology
CF-DETR augments the standard transformer architecture with modules designed to provide local precision to a global model. The core of the approach is the Coarse-to-Fine (CF) Decoder Layer, which splits processing into a "Coarse" sub-layer for global context and a "Fine" sub-layer for precise refinement.
Key components include:
- Transformer Enhanced FPN (TEF): Injects high-level, non-local information from the Transformer Encoder back into lower-level feature maps.
- Adaptive Scale Fusion (ASF): Uses object queries to adaptively weight and fuse features from multiple scales of the pyramid.
- Local Cross-Attention (LCA): Employs depth-wise convolution to focus attention on a small neighborhood around predicted boxes, enabling the model to "zoom in" on boundaries.
The Localization Gap
A pivotal insight of this work is the "localization gap." Analysis shows that vanilla DETR actually outperforms many traditional detectors on small objects at very low Intersection-over-Union (IoU) thresholds (e.g., 10%). This suggests that the global attention mechanism is highly effective at identifying the presence of objects, but lacks the local spatial information required to refine bounding boxes at higher IoU requirements.
Experimental Results
The model was evaluated on the MS COCO 2017 dataset. CF-DETR demonstrates significant improvements in both accuracy and efficiency:
- Accuracy: Reached 47.8 AP with a ResNet-50 backbone, outperforming vanilla DETR (42.0 AP) and Deformable DETR (46.2 AP).
- Small Object Detection: $AP_s$ improved dramatically from 20.5 to 31.2.
- Convergence: Achieved state-of-the-art results in 36 epochs, compared to the 500 epochs required by the original DETR.
- Scalability: Using a ResNeXt-101 backbone and test-time augmentation, the model achieves 53.0 AP.
Discussion and Conclusion
While CF-DETR significantly improves training speed and small object accuracy, it introduces additional architectural complexity. The inclusion of RoI Align and FPN-like structures adds computational overhead, resulting in an inference speed of 14–18 FPS on a V100.
The success of CF-DETR suggests that the future of end-to-end detection lies in hybridizing global transformer attention with local refinement techniques. Future work may explore replacing coarse layers with more advanced attention mechanisms or integrating more sophisticated local operations like deformable convolutions.
Summary updated