OnlineTAS: An Online Baseline for Temporal Action Segmentation

Guodong Ding, Angela Yao, 清 田中

TL;DR

This paper introduces OnlineTAS, the first online baseline for temporal action segmentation, enabling real-time video understanding without future information. It achieves state-of-the-art performance by employing an adaptive memory bank and a context-aware feature augmentation module to effectively capture temporal context in an online setting, addressing the common issue of over-segmentation.

What Problem Does This Paper Address?

Temporal Action Segmentation (TAS) aims to identify and label actions within untrimmed videos, which often depict procedural activities composed of multiple steps. Traditional TAS methods operate offline, meaning they have access to the entire video sequence for analysis. However, many real-world applications, such as live surveillance or interactive systems, require online processing, where decisions must be made as the video unfolds, without knowledge of future frames. This online setting presents significant challenges: efficiently processing a continuous stream of data, retaining relevant temporal context over extended periods, and mitigating over-segmentation (where a single action is incorrectly split into multiple segments), which is exacerbated by limited temporal information at the start of an action. Existing online action detection methods are not directly applicable to segmentation due to differences in task requirements and how they handle temporal dependencies.

Methods

The core of OnlineTAS is a novel framework built upon a single-stage Temporal Convolutional Network (TCN) with causal convolutions. The key innovations are:

  • Adaptive Memory Bank: This component stores both short-term (recent clip) and long-term (accumulated past clips) temporal context. It dynamically updates to prioritize relevant information, allowing the model to adapt to changing dynamics within the video.
  • Context-Aware Feature Augmentation (CFA) Module: This module injects the temporal context from the memory bank into the current frame features. It uses a combination of GRU, self-attention, and Transformer decoders to create context-enhanced representations, improving the model's understanding of the current frame within its temporal surroundings.
  • Post-processing for Over-segmentation: A simple yet effective post-processing step is introduced to smooth out predictions. It leverages confidence scores and a minimum segment length threshold to merge over-segmented predictions, significantly improving segmental metrics.

The framework is trained end-to-end on a clip basis, using a modified cross-entropy loss that encourages smooth transitions between action segments. Two inference modes are proposed: "online" (stride 1, frame-level prediction) and "semi-online" (stride equal to clip length, preserving all predictions within a clip).

Datasets

The paper evaluates OnlineTAS on three standard temporal action segmentation benchmarks:

  • GTEA [12]: Contains videos of kitchen activities.
  • 50Salads [38]: Features videos of food preparation.
  • Breakfast [18]: A larger dataset with diverse activities and actions.

The authors use common I3D features as input for these datasets.

Key Insights and Ideas

  • The critical need for online temporal context: The paper strongly emphasizes that simply adapting offline models or online detection models to segmentation is insufficient. Effectively capturing and utilizing temporal context as it arrives is paramount for accurate online action segmentation.
  • Adaptive memory is key to long-term context: The adaptive memory bank's ability to dynamically balance short-term and long-term information is crucial. This allows the model to retain relevant historical information without being overwhelmed by irrelevant details, a common challenge in long videos.
  • Post-processing is essential for practical performance: While the core model shows promise, the severe over-segmentation issue in online settings necessitates a dedicated post-processing step. This step, though simple, dramatically improves segmental metrics, making the online predictions much more usable.
  • Semi-online inference offers a performance-accuracy trade-off: The semi-online inference mode, which processes clips and preserves all predictions within them, consistently outperforms the frame-by-frame online mode, highlighting the benefit of having slightly more temporal information available per prediction step.

Key Results

  • State-of-the-art online performance: OnlineTAS achieves state-of-the-art results on the evaluated benchmarks for online temporal action segmentation.
  • Significant improvements over baselines: On 50Salads, the proposed approach (with post-processing) shows a substantial boost of 5.7% in Accuracy and 9.2% in Edit score compared to a baseline causal TCN.
  • Mitigation of over-segmentation: The post-processing step dramatically improves segmental metrics. For example, on Breakfast, the F1 score at 50% IoU is nearly tripled after post-processing.
  • Competitive with offline methods: In some cases, the semi-online version of OnlineTAS with post-processing surpasses certain offline methods on specific metrics, demonstrating the effectiveness of the online approach.
  • Runtime efficiency: The model achieves high inference speeds, with up to 238.1 FPS using pre-computed I3D features and 33.8 FPS for the full pipeline including optical flow and feature extraction.

Limitations and Future Work

  • Handling interrupted actions: The paper acknowledges that handling scenarios where subjects abruptly switch actions, leaving ongoing actions unfinished, remains a challenge.
  • Managing infinitely long videos: Effectively preserving long-form history within a fixed memory budget for streaming, potentially infinitely long videos, is identified as a critical future research direction.
  • Dataset bias: The current evaluation is primarily on cooking videos. The authors suggest that handling more diverse and real-world videos might present additional challenges.

The authors propose exploring stronger backbones and improved temporal context modeling to further address these limitations.

Summary updated

BibTeX
@inproceedings{ding2024onlinetas,
  author = {Guodong Ding and Angela Yao and 清 田中},
  title = {{OnlineTAS: An Online Baseline for Temporal Action Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=bkLetzd97M},
}
Figure