End-to-End Video Object Detection with Spatial-Temporal Transformers

Lu He, Qianyu Zhou, Xiangtai Li, Li Niu, Guangliang Cheng, Xiao Li, Wenxuan Liu, Yunhai Tong, Lizhuang Ma, Liqing Zhang

TL;DR

This paper introduces TransVOD, a novel end-to-end Transformer-based framework for video object detection (VOD). By leveraging spatial-temporal Transformers, TransVOD effectively integrates information across frames, eliminating the need for complex hand-crafted components like optical flow or recurrent networks, and simplifying the VOD pipeline.

What Problem Does This Paper Address?

Traditional object detection methods, while successful on static images, struggle with the complexities of video, such as motion blur, occlusion, and appearance changes. Existing video object detection (VOD) approaches often rely on either post-processing techniques to link detections across frames or complex feature aggregation methods that incorporate temporal information. These methods frequently involve hand-designed components like optical flow estimation, recurrent neural networks (RNNs), or relation networks, making the overall pipeline cumbersome and computationally expensive. The paper aims to address this by proposing a streamlined, end-to-end VOD framework that leverages the power of Transformers to handle temporal dependencies without these intricate additions.

Methods

TransVOD builds upon the success of DETR and Deformable DETR, adapting their Transformer-based architecture for video. The core innovation lies in the Temporal Transformer module, which integrates information across multiple frames. This module comprises three key components:

  • Temporal Deformable Transformer Encoder (TDTE): This component efficiently aggregates spatial features from multiple frames using a deformable attention mechanism. This allows it to focus on relevant spatial-temporal information while avoiding the computational overhead of attending to every pixel.
  • Temporal Query Encoder (TQE): This module is designed to fuse object queries across different frames. It uses a coarse-to-fine strategy to select and aggregate relevant object queries from reference frames, enhancing the representation of objects in the current frame.
  • Temporal Deformable Transformer Decoder (TDTD): This decoder takes the outputs from TDTE and TQE to produce the final detection results for the current frame. It uses deformable attention to combine the aggregated memory encodings and temporal queries.

The overall architecture uses a shared CNN backbone for feature extraction, followed by a Spatial Transformer (based on Deformable DETR) for each frame to produce spatial object queries and feature memories. These are then processed by the Temporal Transformer components to achieve video object detection.

Datasets

The paper uses the ImageNet VID dataset for training and evaluation. This dataset is a standard benchmark for video object detection.

Key Insights and Ideas

  • Transformers are well-suited for VOD: The paper demonstrates that the Transformer architecture, with its ability to model long-range dependencies, is a powerful tool for capturing temporal information in videos, a crucial aspect for VOD.
  • Simplifying the VOD pipeline: By integrating temporal modeling directly into the Transformer architecture, TransVOD eliminates the need for separate optical flow modules, RNNs, or complex post-processing steps like Seq-NMS, leading to a cleaner and more efficient system.
  • Object queries as temporal links: The concept of object queries, introduced in DETR, is extended. These queries, which represent instances within a frame, are effectively linked and refined across frames by the Temporal Query Encoder (TQE), allowing for better temporal consistency.
  • Deformable attention for efficiency: The use of deformable attention in both the encoder (TDTE) and decoder (TDTD) allows the model to efficiently sample relevant spatial-temporal locations, reducing computational cost compared to standard attention mechanisms.

Key Results

  • TransVOD achieves significant improvements over its single-frame baseline (Deformable DETR), boosting performance by 3-4% mAP on the ImageNet VID dataset.
  • With a ResNet-50 backbone, TransVOD achieves 79.9% mAP, outperforming previous state-of-the-art methods by a large margin.
  • Using a stronger ResNet-101 backbone, TransVOD reaches 81.9% mAP, surpassing many existing methods.
  • The model achieves comparable results to state-of-the-art methods without any post-processing, highlighting its end-to-end nature.
  • Qualitative results show TransVOD's ability to handle challenging scenarios like motion blur and occlusion more effectively than single-frame detectors.

Limitations and Future Work

  • Ablation studies suggest optimal layer counts: The paper includes ablation studies that indicate specific numbers of layers for TQE (5), TDTE (1), and TDTD (1) yield the best results. This suggests that deeper temporal modules might not always be beneficial.
  • Reference frame selection: The paper explores the impact of the number of reference frames, finding that performance stabilizes around 8 frames. Further research could investigate more adaptive or efficient ways to select relevant frames.
  • Future work: The authors hope their work provides a new perspective for VOD and suggests that similar Transformer-based approaches could be applied to other video understanding tasks.

Summary updated

Abstract

Recently, DETR and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, an end-to-end video object detection model based on a spatial-temporal Transformer architecture. The goal of this paper is to streamline the pipeline of VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow, recurrent neural networks, relation networks. Besides, benefited from the object query design in DETR, our method does not need complicated post-processing methods such as Seq-NMS or Tubelet rescoring, which keeps the pipeline simple and clean. In particular, we present temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal Transformer consists of three components: Temporal Deformable Transformer Encoder (TDTE) to encode the multiple frame spatial details, Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3%-4% mAP) on the ImageNet VID dataset. TransVOD yields comparable results performance on the benchmark of ImageNet VID. We hope our TransVOD can provide a new perspective for video object detection.
BibTeX
@inproceedings{he2021end,
  author = {Lu He and Qianyu Zhou and Xiangtai Li and Li Niu and Guangliang Cheng and Xiao Li and Wenxuan Liu and Yunhai Tong and Lizhuang Ma and Liqing Zhang},
  title = {{End-to-End Video Object Detection with Spatial-Temporal Transformers}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=X96Tv2c9cj},
}
Figure