Action Completion: A Temporal Model for Moment Detection

Farnoosh Heidarivincheh, Majid Mirmehdi, Dima DamenarXiv 2018 · arXiv

TL;DR

This paper introduces the task of completion moment detection: identifying the exact frame in a video where an action’s goal is achieved (e.g., the moment a ball is caught or a candle is extinguished). The authors propose a joint classification-regression recurrent model that uses per-frame "votes" to robustly locate this moment or identify if the action was never completed.

What Problem Does This Paper Address?

Traditional computer vision tasks like action recognition focus on what is happening (e.g., "someone is drinking"), while action localization identifies the start and end of an attempt. However, these tasks often ignore whether the action's aim was actually achieved.

For instance, in the action "drinking," localization might mark the start when the glass is lifted and the end when it's put down. The "completion moment," however, is the subtle point when the person actually consumes the beverage. Detecting this is crucial for applications like human-robot collaboration (where a robot needs to react to a finished task) or assisted living (detecting if an elderly person failed to complete a safety-critical action like turning off an oven).

Methods

The authors propose a Convolutional-Recurrent Neural Network (C-RNN) architecture designed to accumulate evidence across a video sequence. The core innovation is the Recurrent Voting Node.

The C-RNN architecture: frames pass through a CNN into an LSTM, which outputs classification and regression votes.

Each frame in a sequence contributes to the final decision through two types of "voting":

  1. Classification Voting ($V_C$): Each frame predicts whether it is "pre-completion" or "post-completion." This helps narrow down the temporal split where the moment resides.
  2. Regression Voting ($V_R$): Each frame predicts its relative temporal distance to the completion moment. This allows for fine-grained localization.

The paper explores four schemes for combining these votes. The most successful, C-R (Classification-pre, Regression-post), uses classification for frames before the event and regression for frames after it. The final sequence-level prediction is the frame that receives the maximum accumulated vote.

Different voting schemes (C-C, R-R, R-C, C-R) for sequence-level prediction.

Datasets

The researchers evaluated their model on 16 actions across three public datasets, providing new "completion moment" annotations for each:

  • HMDB: 5 actions (catch, drink, pick, pour, throw) across 494 sequences.
  • UCF101: 5 actions (basketball, blowing candles, frisbee catch, pole vault, soccer penalty) across 650 sequences.
  • RGBD-AC: 6 actions (switch, plug, open, pull, pick, drink) across 414 sequences. This dataset is particularly interesting as it includes "incomplete" attempts, such as a person trying to open a locked drawer.
Figure 1: Annotation of completion moment: Two examples per action. Pre-completion frames are bordered in orange and post-completion in purple. From Top: HMDB pick, UCF101 blowing candles, RGBD-AC switch (one complete sequence and one incomplete).

Key Insights and Ideas

  • Asymmetric Information: A key "aha moment" was finding that pre-completion frames are poor at predicting the exact time remaining until completion, but are very good at simply stating "it hasn't happened yet." Conversely, once the completion moment has passed, the visual evidence (e.g., the ball is now in the hand) makes regression much more accurate. This is why the C-R hybrid scheme outperformed pure regression or pure classification.
  • Global from Local: By treating every frame as a voter rather than just trying to classify the "perfect" frame, the model becomes robust to temporary occlusions or ambiguous visual cues.
  • Handling Incompletion: By extending the voting range to a "T+1" index, the model can naturally represent and detect sequences where the action is attempted but never finished.

Key Results

The proposed C-R method significantly outperformed baselines across all datasets:

  • Temporal Accuracy: The completion moment was detected within 1 second (30 frames) in 89% of all tested sequences and within 0.5 seconds in 74% of sequences.
  • Exact Hits: The model identified the exact ground-truth frame in 30.4% of cases.
  • Robustness: The model successfully distinguished between complete and incomplete actions, achieving high accuracy even in challenging sports sequences with high motion blur.
Qualitative results showing the model's predictions vs. ground truth for various actions.

Limitations and Future Work

  • End-to-End Training: Due to hardware limitations, the authors used a two-stage approach (extracting CNN features first, then training the LSTM). They note that true end-to-end training remains a challenge for future work.
  • Untrimmed Videos: The current model assumes the action is already temporally segmented (i.e., the video starts near the beginning of the attempt). Extending this to long, untrimmed videos is a primary goal for future research.
  • Multiple Moments: The current formulation looks for a single completion moment. Future iterations could aim to detect multiple completion moments in complex, multi-step tasks.

Summary updated

Abstract

We introduce completion moment detection for actions - the problem of locating the moment of completion, when the action's goal is confidently considered achieved. The paper proposes a joint classification-regression recurrent model that predicts completion from a given frame, and then integrates frame-level contributions to detect sequence-level completion moment. We introduce a recurrent voting node that predicts the frame's relative position of the completion moment by either classification or regression. The method is also capable of detecting incompletion. For example, the method is capable of detecting a missed ball-catch, as well as the moment at which the ball is safely caught. We test the method on 16 actions from three public datasets, covering sports as well as daily actions. Results show that when combining contributions from frames prior to the completion moment as well as frames post completion, the completion moment is detected within one second in 89% of all tested sequences.
BibTeX
@misc{heidarivincheh2018action,
  author = {Farnoosh Heidarivincheh and Majid Mirmehdi and Dima Damen},
  title = {{Action Completion: A Temporal Model for Moment Detection}},
  year = {2018},
  eprint = {1805.06749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.06749},
}

Loading related content...

Figure