Abstract
Traditional video object detection (VOD) systems rely on complex, multi-stage pipelines involving optical flow, recurrent neural networks, and hand-crafted post-processing. TransVOD introduces the first fully end-to-end VOD framework based on Transformers. By leveraging a unified spatial-temporal architecture, TransVOD simplifies the detection pipeline while achieving state-of-the-art performance on standard benchmarks. This paper details the architecture of the Temporal Transformer and demonstrates its effectiveness in handling common video challenges such as motion blur and occlusion.
Introduction
Object detection in video sequences presents unique challenges compared to static image detection. Temporal artifacts—including motion blur, video defocus, part occlusion, and rare object poses—frequently cause frame-level detectors to produce inconsistent results, often referred to as "flicker." To mitigate these issues, the computer vision community has historically developed "over-sophisticated" pipelines. These typically integrate heavy components such as optical flow networks for feature warping, Recurrent Neural Networks (RNNs) for state maintenance, and complex post-processing techniques like Seq-NMS or Tubelet rescoring to link detections across time.
While effective, these hand-crafted components make the VOD pipeline difficult to optimize, computationally expensive, and messy to implement. TransVOD seeks to resolve these inefficiencies by reformulating VOD as a sequence-to-sequence decoding problem, building upon the success of the Detection Transformer (DETR) paradigm.
The TransVOD Architecture
TransVOD is built upon Deformable DETR, an evolution of the original Transformer detector that offers faster convergence and higher computational efficiency. The core contribution of TransVOD is the Temporal Transformer, which replaces traditional temporal modeling components with three integrated modules designed to aggregate information across frames.
1. Temporal Deformable Transformer Encoder (TDTE)
The TDTE is responsible for aggregating "feature memories" from multiple video frames. To avoid the quadratic memory costs associated with standard self-attention in video, the TDTE utilizes deformable attention. This mechanism samples only a small set of key points across the temporal dimension, effectively denoising the background and focusing on relevant spatial details across the sequence.
2. Temporal Query Encoder (TQE)
In DETR-based architectures, "object queries" are learnable vectors that probe the feature map for potential objects. TransVOD’s TQE links these queries across time using a coarse-to-fine strategy. It selects high-confidence object queries from reference frames and fuses them with the current frame's queries. This allows the model to maintain a "temporal memory" of an object, ensuring detection stability even if the object is partially obscured or blurred in a specific frame.
3. Temporal Deformable Transformer Decoder (TDTD)
The final stage of the pipeline, the TDTD, receives the fused feature memories from the TDTE and the enhanced temporal queries from the TQE. It produces the final bounding boxes and class scores for the current frame. Because the system is end-to-end, it outputs a sparse set of predictions, entirely removing the need for Non-Maximum Suppression (NMS) or external tubelet linking.
Experimental Results
Datasets and Training
The authors evaluated TransVOD on the ImageNet VID dataset, the industry standard for video object detection. The model was trained using a combination of ImageNet VID and ImageNet DET (still images) to provide a robust foundation for both spatial and temporal recognition. Evaluation was conducted on the 555 validation videos of ImageNet VID, covering 30 distinct object categories.
Performance Metrics
TransVOD established new state-of-the-art benchmarks for VOD without the need for post-processing:
- Accuracy: The model achieved 79.9% mAP with a ResNet-50 backbone and 81.9% mAP with a ResNet-101 backbone.
- Baseline Improvement: This represents a significant 3-4% mAP increase over the single-frame Deformable DETR baseline, proving the value of the temporal modules.
- Comparative Analysis: TransVOD outperformed previous leading models, including MEGA (77.3%) and FGFA (74.0%), despite those models utilizing significantly more complex, flow-based pipelines.
Qualitative analysis further demonstrates that TransVOD is remarkably robust to rare poses and heavy motion blur, scenarios where single-frame detectors typically fail.
Discussion and Key Insights
One of the most significant insights of TransVOD is the treatment of object queries as temporal anchors. By linking queries across frames, the model naturally maintains object identity without requiring explicit tracking algorithms or optical flow. Furthermore, the use of deformable attention allows the model to aggregate information from up to 15 frames simultaneously without the prohibitive memory costs of standard Transformers.
By removing optical flow networks—which can add upwards of 100 million parameters—TransVOD achieves a more efficient parameter count (58.9M for ResNet-101) while delivering superior accuracy. This suggests that simplicity and end-to-end optimization are more effective than hand-crafted complexity for video tasks.
Conclusion and Future Work
TransVOD represents a major shift in video object detection, proving that a unified Transformer architecture can outperform specialized, multi-stage pipelines. While the current model operates on a fixed temporal window (typically 9 to 15 frames), future research may focus on extending this to long-form streaming video. Additionally, the spatial-temporal Transformer approach holds promise for related tasks such as Video Instance Segmentation (VIS) and Multi-Object Tracking (MOT), where maintaining object identity over time is paramount.
Summary updated