Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videos

Reza Ghoddoosian, Saif Sayed, Vassilis AthitsosWACV 2021

TL;DR

This paper tackles the challenging problem of aligning video-level action labels to specific time segments within a video, even when only the order of actions is known (weak supervision). They introduce a novel "Duration Network" that leverages visual features to predict action durations and a "Segment-Level Beam Search" to efficiently find the best alignment, achieving state-of-the-art results on benchmark datasets.

What Problem Does This Paper Address?

Recognizing and segmenting actions in videos is crucial for many applications, from surveillance to human-computer interaction. While fully supervised methods (where every frame is labeled) achieve good results, they require expensive, frame-level annotations. "Weak supervision" offers a more scalable alternative, providing only the ordered sequence of actions present in a video, without precise start and end times. The core challenge in weakly supervised action alignment is accurately predicting the duration of each action, which is essential for segmenting the video correctly. Existing methods often rely on simple statistical models (like Poisson distributions) for duration prediction, which ignore valuable information present in the video's visual content.

Methods

The paper proposes two main novel components:

  • Duration Network (DurNet): This network is designed to predict the remaining duration of an ongoing action. Unlike previous methods that only consider the action type, DurNet takes into account:
  • Local visual features: The actual visual content of a short temporal window around the current frame.
  • Elapsed duration: How long the current action has already been ongoing.
  • Action type: The specific action being performed. DurNet uses a bi-directional LSTM to process these inputs and outputs a probability distribution over discretized duration bins. The duration bins are made verb-dependent to account for varying action lengths.

  • Segment-Level Beam Search: This algorithm efficiently finds the optimal alignment of actions to video segments. Instead of evaluating every frame as a potential start/end point (like frame-level beam search), it considers potential segment boundaries only at the beginning of each segment. This significantly reduces computational complexity. The search leverages the DurNet's duration predictions and an "Action Selector Network" (which predicts the action label based on verb and object) to explore candidate alignments and select the most probable sequence.

Datasets

The proposed method is evaluated on two standard weakly-supervised action alignment datasets:

  • Breakfast Dataset (BD): Contains untrimmed instructional videos of cooking activities with 48 action labels. It's characterized by an expected scenario and context, and a relatively low percentage of background frames.
  • Hollywood Extended Dataset (HED): Features videos from movies with 17 action labels. This dataset is more challenging due to random action sequences, a high percentage of background frames, and scene cuts.

Key Insights and Ideas

  • Visual context is crucial for duration prediction: The paper demonstrates that incorporating visual features into duration prediction significantly improves alignment accuracy compared to purely statistical models. This is because visual cues can indicate how an action is progressing and its likely future duration (e.g., a person just picking up an orange versus actively squeezing it).
  • Segment-level beam search is more efficient: By focusing on segment beginnings rather than every frame, the proposed beam search drastically reduces computational cost, making it more practical for long videos.
  • Verb-object decomposition aids generalization: Decomposing actions into verbs and objects allows the model to leverage shared visual characteristics of verbs across different actions, which is particularly helpful in weakly supervised settings where frame-level ground truth might be noisy.
  • Longer videos benefit more from adaptive duration: The proposed method shows improved robustness on longer videos, where its ability to adapt duration predictions based on context is more impactful than fixed-duration models.

Key Results

  • The proposed method achieves state-of-the-art results on the Breakfast and Hollywood Extended datasets in certain metrics.
  • It outperforms existing methods like TCFPN, NNViterbi, and CDFL, particularly in frame-level accuracy without background frames (acc-bg), which is crucial for datasets with significant background content like HED.
  • The DurNet improves alignment accuracy by over 1% compared to using a Poisson duration model.
  • The Segment-Level Beam Search is significantly more efficient, with a time complexity of O(N) (where N is the number of segments) compared to O(T) or O(T^2) for frame-level Viterbi algorithms (where T is the number of frames).

Limitations and Future Work

  • Background frame alignment: The paper acknowledges that aligning background segments remains challenging. This is attributed to the lack of defined structure in background scenes and the inherent difficulty for humans to predict the duration of motionless or non-eventful periods.
  • DurNet performance dependency on training data: The performance of the DurNet, like other neural networks, is shown to be dependent on the size of the training data.
  • Pseudo-ground truth initialization: The training strategy relies on pseudo-ground truth generated by existing weakly supervised methods. The quality of this initialization can impact the final performance.
  • Future work: The authors suggest exploring more sophisticated action selector components and potentially incorporating more explicit temporal modeling beyond the current LSTM architecture.

Summary updated

BibTeX
@inproceedings{ghoddoosian2021action,
  author = {Reza Ghoddoosian and Saif Sayed and Vassilis Athitsos},
  title = {{Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure