Exploring frame segmentation networks for temporal action localization

Ke Yang, Xiaolong Shen, Peng Qiao, Shijie Li, Dongsheng Li, Yong Dou

TL;DR

This paper introduces the Frame Segmentation Network (FSN), a novel architecture that combines 2D spatial CNNs with 1D temporal CNNs to achieve precise temporal action localization in videos. FSN enables dense, frame-level predictions by effectively modeling spatio-temporal information, and can also be adapted for weakly supervised scenarios.

What Problem Does This Paper Address?

Temporal action localization is a crucial task in computer vision, aiming to identify not only what action is happening in a video but also when it occurs, including its precise start and end times. While many methods exist, accurately predicting these temporal boundaries remains a challenge. Existing approaches often rely on pre-defined action proposals or sliding windows, which can be inefficient and limit precision. Furthermore, most methods require strong supervision (e.g., frame-level annotations), which is labor-intensive to obtain. This paper addresses the need for a more precise, efficient, and adaptable method for temporal action localization, including a weakly supervised variant.

Methods

The core of this work is the Frame Segmentation Network (FSN). FSN's architecture is designed to capture both spatial and temporal information effectively. It comprises two main components:

  • 2D Spatial CNNs: These networks process individual video frames to extract rich semantic features, capturing the appearance of objects and scenes. The paper suggests using pre-trained networks like VGG or ResNet for this purpose, leveraging their strong performance on image recognition tasks.
  • 1D Temporal CNNs: Stacked on top of the 2D CNNs, these networks take a sequence of feature vectors (one for each frame) and model the temporal context. This allows FSN to understand how actions evolve over time and make dense predictions at a frame-level resolution. To enlarge the temporal receptive field without increasing parameters significantly, the authors employ dilated convolutions in the temporal CNN.

FSN is trained end-to-end, allowing for joint optimization of spatial and temporal components. The paper also proposes a Weakly Supervised FSN (WFSN) variant. WFSN adapts FSN to scenarios where only video-level labels (indicating the presence of an action in a video, but not its exact temporal boundaries) are available. This is achieved by incorporating a temporal pooling layer (specifically, Global Max Pooling, GMP, which is found to be more effective than Global Average Pooling for this task) before the final classification, allowing the network to infer temporal localization from video-level cues.

Datasets

The proposed methods are evaluated on the THUMOS'14 dataset. This dataset is described as challenging and contains 101 action classes. For training, the authors also utilize the UCF101 dataset. The THUMOS'14 challenge specifically focuses on temporal action detection and involves 20 action classes. The dataset is split into training, validation, and test sets, with specific numbers of videos and action instances provided for each.

Key Insights and Ideas

  • Decomposition of 3D CNNs: The paper argues that while 3D CNNs are suitable for spatio-temporal modeling, their large parameter count and data requirements make them less practical. FSN offers an alternative by decomposing this into separate 2D spatial and 1D temporal CNNs, achieving similar goals with greater efficiency.
  • Dense Frame-Level Predictions: Moving beyond segment-level proposals, FSN's ability to make dense, frame-level predictions is crucial for precise temporal boundary localization. This fine-grained prediction capability is a key differentiator.
  • Effectiveness of Temporal CNNs: The experiments demonstrate that incorporating a temporal CNN significantly improves performance over methods that only consider spatial information (single-frame CNNs) or rely on less sophisticated temporal modeling.
  • Weak Supervision is Viable: The WFSN variant shows that it's possible to achieve competitive temporal action localization results even with limited supervision (video-level labels), which has significant implications for reducing annotation costs.
  • GMP Outperforms GAP for Weak Supervision: The finding that Global Max Pooling (GMP) is more effective than Global Average Pooling (GAP) for temporal pooling in the weakly supervised setting is an important practical insight. This suggests that focusing on the most discriminative temporal moments is more beneficial than averaging over the entire video.

Key Results

  • Frame-Level Localization: FSN achieves a mean Average Precision (mAP) of 53.5% on the THUMOS'14 dataset for frame-level action localization, significantly outperforming previous state-of-the-art methods like CDC (44.4%) and MultiLSTM (41.3%).
  • Segment-Level Localization: For segment-level action localization on THUMOS'14, FSN achieves a competitive mAP of 51.8% at an IoU threshold of 0.7, and 32.1% at an IoU threshold of 0.5.
  • Weakly Supervised Performance: WFSN demonstrates strong performance in weakly supervised settings, achieving an mAP of 54.9% at an IoU threshold of 0.1 and 38.9% at an IoU threshold of 0.5 on THUMOS'14. This performance is comparable to or better than several strongly supervised methods.
  • Ablation Study: An ablation study shows that removing the temporal CNN component (FSN w/o Temporal CNN) leads to a drop in frame-level mAP from 47.5% to 42.5% (for the RGB stream), highlighting the necessity of temporal modeling.

Limitations and Future Work

  • Temporal Receptive Field: The paper notes that while dilated convolutions help, the temporal receptive field might still not be large enough in some cases, potentially leading to false positives. They suggest exploring techniques like multiple resolutions in the future.
  • Data Annotation for Weak Supervision: For WFSN, the authors observe that the lack of temporal boundary information during training can lead to false positives. They suggest that a small amount of annotated data might still be beneficial.
  • "Whole Action" Concept: The observation that WFSN predictions are often shorter than FSN predictions, similar to weakly supervised object detection, suggests a need to incorporate mechanisms to learn the concept of a "whole action" rather than just discriminative parts.
  • Future Work: The authors implicitly suggest exploring more advanced temporal modeling techniques and potentially incorporating multi-resolution features to address the limitations.

Summary updated

Abstract

Temporal action localization is an important task of computer vision. Though many methods have been proposed, it still remains an open question how to predict the temporal location of action segments precisely. Most state-of-the-art works train action classifiers on video segments pre-determined by action proposal. However, recent work found that a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. In this paper, we propose a Frame Segmentation Network (FSN) that places a temporal CNN on top of the 2D spatial CNNs. Spatial CNNs are responsible for abstracting semantics in spatial dimension while temporal CNN is responsible for introducing temporal context information and performing dense predictions. The proposed FSN can make dense predictions at frame-level for a video clip using both spatial and temporal context information. FSN is trained in an end-to-end manner, so the model can be optimized in spatial and temporal domain jointly. We also adapt FSN to use it in weakly supervised scenario (WFSN), where only video level labels are provided when training. Experiment results on public dataset show that FSN achieves superior performance in both frame-level action localization and temporal action localization.
BibTeX
@article{yang2019exploring,
  author = {Ke Yang and Xiaolong Shen and Peng Qiao and Shijie Li and Dongsheng Li and Yong Dou},
  title = {{Exploring frame segmentation networks for temporal action localization}},
  journal = {J. Vis. Commun. Image Represent.},
  year = {2019},
  url = {https://openreview.net/forum?id=nPyjUrix6YV},
}

Loading related content...

Figure