Introduction
Video object detection (VOD) presents unique challenges compared to static image detection. Issues such as motion blur, partial occlusion, and unusual object poses—collectively termed "appearance deterioration"—frequently degrade the quality of individual frames. To overcome these, modern VOD frameworks aggregate temporal information across multiple frames to refine feature representations.
Current research typically follows two distinct paths. Transformer-based methods excel at capturing long-range global dependencies but often overlook fine-grained local spatial-temporal details. Conversely, CNN-based methods are adept at local feature extraction but struggle with the "big picture" temporal context. TGBFormer (Transformer-GraphFormer Blender Network) bridges this gap by hybridizing these approaches, adaptively blending global and local features to achieve state-of-the-art performance.
Architecture and Methodology
The TGBFormer architecture is designed to resolve the global-local trade-off through a three-pronged approach: the Spatial-Temporal Transformer Module (STTM), the Spatial-Temporal GraphFormer Module (STGM), and a Global-Local Feature Blender.
Spatial-Temporal Transformer Module (STTM)
The STTM serves as the global context aggregator. Utilizing multi-head self-attention (MHSA), it models long-range dependencies across both spatial dimensions within a frame and temporal dimensions across the video sequence. This allows the network to maintain a consistent understanding of object identity and location even when visual cues are temporarily weakened.
Spatial-Temporal GraphFormer Module (STGM)
The STGM acts as the local specialist. It represents image features as nodes in a graph. To prevent the noise associated with fully connected graphs, the STGM employs a Graph Pruning mechanism that removes weak connections, focusing on relevant local structures. Dynamic Graph Convolutions then aggregate features from these pruned neighborhoods, capturing the fine-grained spatial-temporal relationships that Transformers might miss.
Global-Local Feature Blender
A critical innovation of TGBFormer is the Blender module. Rather than simple addition or concatenation, this module employs a learnable weighting system to dynamically fuse global and local features. By analyzing the input features, the Blender decides the optimal ratio of global context to local detail for each specific frame, ensuring robustness against varying scene complexities.
Sequence-wise Detection
To maintain high inference speeds, TGBFormer adopts a parallel sequence-wise detection strategy. Instead of sequential frame processing, it handles chunks of frames (typically $N=25$) simultaneously. This parallelization allows the model to achieve real-time performance without sacrificing the depth of its feature aggregation.
Experimental Results
TGBFormer was evaluated on the ImageNet VID and DET benchmarks, which are standard for assessing video object detection performance.
Accuracy and Efficiency
As shown in the results table, TGBFormer achieves a new state-of-the-art 86.5% mAP using a ResNet-101 backbone, outperforming recent competitors like CDANet and ClipVID. When paired with a Swin-B backbone, the performance reaches an impressive 90.3% mAP.
Crucially, this accuracy does not come at the cost of speed. TGBFormer maintains an inference rate of 41.0 FPS (24.3ms per frame) on a Tesla A100 GPU. This balance of high precision and real-time capability makes it highly suitable for practical applications such as autonomous driving and surveillance.
Key Insights from Ablation Studies
Ablation experiments highlighted the importance of the Blender module, which provided a 1.1% mAP improvement over standard concatenation. Furthermore, the study of graph depth revealed that two layers of dynamic graph convolution represent a "sweet spot," preventing the over-smoothing issues common in deeper graph architectures.
Conclusion
TGBFormer successfully addresses the inherent limitations of using either Transformers or Graphs in isolation for video object detection. By treating global and local feature extraction as complementary tasks and introducing a dynamic blending mechanism, the framework sets a new benchmark for both accuracy and efficiency in the field. Future work may explore lighter backbones to bring this high-performance detection to edge devices.
Summary updated