Abstract
Recently, end-to-end detectors based on transformer (DETRs) have made remarkable progress. However, their high computational cost still limits the performance of the DETRs series as real-time object detectors. In order to solve this problem, we introduce the Van-DETR model, which enhances the first real-time end-to-end object detector, RT-DETR. Specifically, we innovatively introduce a new, more lightweight backbone—VanillaNet, replacing the former backbone—ResNet. To address its weak nonlinearity and poor local analysis, we combine large kernel convolutions with small kernel convolutions to integrate global and local information, significantly enhancing feature extraction capabilities. Secondly, in the hybrid encoder, we cascade group process the features extracted by the backbone and design a gated linear unit with a star-shaped connection for intra-scale feature interaction. During the cross-scale feature fusion stage, we propose a high-low frequency feature fusion module with strong feature representation capabilities. To verify the effectiveness of the model, we conduct experiments on two public object detection datasets—visdrone dataset and a people dataset from roboflow. Experimental results show that the proposed Van-DETR model achieves MAP<sub>50</sub> of 0.471 and 0.730 on two object detection datasets, respectively, representing improvements of 4.5% and 2.8% over the original RT-DETR model.
Introduction
Object detection is a cornerstone of computer vision, with applications ranging from autonomous driving to drone-based surveillance. While Transformer-based detectors (DETRs) have eliminated the need for hand-crafted components like Non-Maximum Suppression (NMS), their deployment in real-time scenarios remains challenging due to high computational demands. RT-DETR was the first to achieve real-time performance, yet it still relies on traditional ResNet backbones and standard attention mechanisms that incur significant memory and computational overhead. Van-DETR addresses these limitations by adopting a minimalist architectural philosophy combined with advanced feature fusion techniques.
Methodology
The Van-DETR architecture introduces several key innovations designed to optimize the efficiency-accuracy trade-off.
VanillaNet Backbone and GLFusion
The heavy ResNet backbone is replaced with VanillaNet, a minimalist architecture that avoids complex shortcuts. To mitigate the "weak nonlinearity" inherent in such a shallow design, the authors introduce Global-Local Fusion (GLFusion). This module integrates global context via large-kernel convolutions with local details from small-kernel convolutions, significantly boosting feature extraction power.
Star-Gated Linear Unit (SGLU)
A critical component of the intra-scale feature interaction is the Star-Gated Linear Unit (SGLU). Leveraging the "Star operation" (element-wise multiplication), this module projects features into a high-dimensional implicit space. Theoretical analysis suggests that multiplication is superior to simple summation for making features linearly separable, allowing the model to classify complex targets more effectively without a massive increase in parameters.
Advanced Attention and Fusion
Van-DETR utilizes Cascaded Group Attention (CGA) to reduce redundancy in multi-head attention by splitting heads into groups and processing them sequentially. Furthermore, the High-Low Frequency Feature Fusion (HLFFF) module treats shallow features as high-frequency (textures) and deep features as low-frequency (semantics), employing a bidirectional mechanism to preserve information across scales.
Experimental Results
The model was evaluated on the VisDrone and Roboflow People datasets.
- VisDrone Dataset: Van5-DETR achieved a 47.1% mAP₅₀, marking a 4.5% improvement over the RT-DETR-r18 baseline.
- People Dataset: The model reached 73.0% mAP₅₀, a 2.8% improvement.
- Efficiency: Van-DETR demonstrated superior speed, reaching 38.1 FPS, which is approximately 7 FPS faster than the original RT-DETR-r18.
Qualitative analysis indicates that Van-DETR is particularly robust in detecting small or camouflaged objects in challenging environments, such as low-light drone footage.
Conclusion
Van-DETR demonstrates that a minimalist backbone, when paired with sophisticated feature fusion and interaction modules like SGLU and HLFFF, can outperform deeper, more complex architectures. By prioritizing efficient mathematical operations and frequency-aware fusion, Van-DETR sets a new benchmark for real-time end-to-end object detection. Future work will explore scaling these techniques to larger model variants and multimodal data integration.
Summary updated