Non-Local Temporal Difference Network for Temporal Action Detection

Yilong He, Xiao Han, Yong Zhong, Lishun Wang

TL;DR

This paper introduces the Non-Local Temporal Difference (NTD) network, a novel approach for temporal action detection in untrimmed videos. By combining chunk convolution for long-range context, multi-scale temporal coordination, and a temporal difference module for motion enhancement, NTD achieves state-of-the-art performance on challenging benchmarks.

What Problem Does This Paper Address?

Understanding human actions within videos is crucial for applications like surveillance and content analysis. Temporal Action Detection (TAD) specifically aims to pinpoint when an action starts and ends, and what action it is, within unedited videos. Existing methods often struggle with:

  • Long-range dependencies: Capturing relationships between frames that are far apart in time is difficult with standard convolutional approaches, which tend to focus on local information.
  • Information degradation: As information travels through many layers of a network, it can weaken, especially over long temporal distances.
  • Varying action durations: Actions can be very short or very long, making it hard for a single model to effectively capture all of them.
  • Redundant information: Adjacent frames in a video often contain very similar information, which can lead to inefficiencies and "local traps" in learning.

Methods

The NTD network tackles these challenges with three novel modules:

  • Chunk Convolution (CC) Module: Instead of processing frames sequentially, this module divides the video into "chunks." It then uses multiple independent convolutional blocks to extract features from neighboring chunks simultaneously. This allows the network to capture information from distant frames without getting stuck in local patterns and helps mitigate redundancy.
  • Multiple Temporal Coordination (MTC) Module: This module addresses the issue of varying action durations. It employs a cascade residual architecture with multiple parallel branches. Each branch processes features at a different temporal receptive field, effectively aggregating multiscale temporal features without adding significant computational overhead.
  • Temporal Difference (TD) Module: This module focuses on enhancing motion and boundary features, which are critical for accurate localization. It adaptively uses temporal attention weights to highlight motion-sensitive locations and boost their features. This helps the network better discern actions, especially when motion is subtle or information has weakened over time.

Datasets

The NTD network was evaluated on two widely used benchmark datasets for temporal action detection:

  • THUMOS-14: This dataset contains videos of daily human activities with a focus on densely distributed and disordered action instances, making it a challenging benchmark.
  • ActivityNet-v1.3: A larger dataset with a wider variety of action categories and longer average action durations.

Key Insights and Ideas

  • Chunking for Long-Range Context: The idea of dividing the video into chunks and processing them in parallel is a clever way to break free from the limitations of purely local convolutions and capture long-range dependencies more effectively.
  • Multi-branch for Multi-Scale: The MTC module's cascade residual architecture elegantly handles the varying durations of actions by creating features with different temporal receptive fields, akin to multi-scale processing in image tasks.
  • Attention for Motion Enhancement: The TD module's use of temporal attention to selectively boost motion-sensitive features is a powerful way to combat information degradation and improve localization accuracy.
  • Synergy of Modules: The paper strongly emphasizes that the combination of CC, MTC, and TD modules is crucial for achieving state-of-the-art results, highlighting the benefit of addressing different aspects of temporal action detection in a unified framework.

Key Results

The NTD network achieved state-of-the-art performance on both benchmark datasets:

  • THUMOS-14: 71.6% [email protected] and 66.8% mAP@avg. This represents a significant improvement over previous methods, particularly outperforming ActionFormer by 6.0% on [email protected].
  • ActivityNet-v1.3: 54.4% [email protected] and 36.3% mAP@avg. This also shows competitive results, with the average mAP being 0.7% higher than the previous state-of-the-art.

Qualitative results (Figure 2) demonstrate the model's ability to correctly localize instances of varying durations, though it notes some imprecision for very short actions due to rapid motion changes and less contextual relevance.

Limitations and Future Work

  • Short Action Imprecision: The authors acknowledge that very short action instances can be challenging to localize precisely. This is attributed to rapid motion changes and a lack of sufficient contextual clues, as well as the sensitivity of IoU metrics to small offsets in boundary predictions.
  • Future Directions: The paper suggests that future work should focus on addressing other video characteristics like overlap, nonlinearity, spatio-temporal correlation, inconsistent motion rates, and sparse boundary points. The core challenge remains designing efficient networks that can effectively model complex temporal relationships while accounting for these inherent video complexities.

Summary updated

Abstract

As an important part of video understanding, temporal action detection (TAD) has wide application scenarios. It aims to simultaneously predict the boundary position and class label of every action instance in an untrimmed video. Most of the existing temporal action detection methods adopt a stacked convolutional block strategy to model long temporal structures. However, most of the information between adjacent frames is redundant, and distant information is weakened after multiple convolution operations. In addition, the durations of action instances vary widely, making it difficult for single-scale modeling to fit complex video structures. To address this issue, we propose a non-local temporal difference network (NTD), including a chunk convolution (CC) module, a multiple temporal coordination (MTC) module, and a temporal difference (TD) module. The TD module adaptively enhances the motion information and boundary features with temporal attention weights. The CC module evenly divides the input sequence into N chunks, using multiple independent convolution blocks to simultaneously extract features from neighboring chunks. Therefore, it realizes the information delivered from distant frames while avoiding trapping into the local convolution. The MTC module designs a cascade residual architecture, which realizes the multiscale temporal feature aggregation without introducing additional parameters. The NTD achieves a state-of-the-art performance on two large-scale datasets, 36.2% mAP@avg and 71.6% [email protected] on ActivityNet-v1.3 and THUMOS-14, respectively.
BibTeX
@inproceedings{he2022non,
  author = {Yilong He and Xiao Han and Yong Zhong and Lishun Wang},
  title = {{Non-Local Temporal Difference Network for Temporal Action Detection}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=Ripy99LI2zZ},
}
Figure