TL;DR
This paper introduces the task of completion moment detection: identifying the exact frame in a video where an action’s goal is achieved (e.g., the moment a ball is caught or a candle is extinguished). The authors propose a joint classification-regression recurrent model that uses per-frame "votes" to robustly locate this moment or identify if the action was never completed.
What Problem Does This Paper Address?
Traditional computer vision tasks like action recognition focus on what is happening (e.g., "someone is drinking"), while action localization identifies the start and end of an attempt. However, these tasks often ignore whether the action's aim was actually achieved.
For instance, in the action "drinking," localization might mark the start when the glass is lifted and the end when it's put down. The "completion moment," however, is the subtle point when the person actually consumes the beverage. Detecting this is crucial for applications like human-robot collaboration (where a robot needs to react to a finished task) or assisted living (detecting if an elderly person failed to complete a safety-critical action like turning off an oven).
Methods
The authors propose a Convolutional-Recurrent Neural Network (C-RNN) architecture designed to accumulate evidence across a video sequence. The core innovation is the Recurrent Voting Node.
Each frame in a sequence contributes to the final decision through two types of "voting":
- Classification Voting ($V_C$): Each frame predicts whether it is "pre-completion" or "post-completion." This helps narrow down the temporal split where the moment resides.
- Regression Voting ($V_R$): Each frame predicts its relative temporal distance to the completion moment. This allows for fine-grained localization.
The paper explores four schemes for combining these votes. The most successful, C-R (Classification-pre, Regression-post), uses classification for frames before the event and regression for frames after it. The final sequence-level prediction is the frame that receives the maximum accumulated vote.
Datasets
The researchers evaluated their model on 16 actions across three public datasets, providing new "completion moment" annotations for each:
- HMDB: 5 actions (catch, drink, pick, pour, throw) across 494 sequences.
- UCF101: 5 actions (basketball, blowing candles, frisbee catch, pole vault, soccer penalty) across 650 sequences.
- RGBD-AC: 6 actions (switch, plug, open, pull, pick, drink) across 414 sequences. This dataset is particularly interesting as it includes "incomplete" attempts, such as a person trying to open a locked drawer.
Key Insights and Ideas
- Asymmetric Information: A key "aha moment" was finding that pre-completion frames are poor at predicting the exact time remaining until completion, but are very good at simply stating "it hasn't happened yet." Conversely, once the completion moment has passed, the visual evidence (e.g., the ball is now in the hand) makes regression much more accurate. This is why the C-R hybrid scheme outperformed pure regression or pure classification.
- Global from Local: By treating every frame as a voter rather than just trying to classify the "perfect" frame, the model becomes robust to temporary occlusions or ambiguous visual cues.
- Handling Incompletion: By extending the voting range to a "T+1" index, the model can naturally represent and detect sequences where the action is attempted but never finished.
Key Results
The proposed C-R method significantly outperformed baselines across all datasets:
- Temporal Accuracy: The completion moment was detected within 1 second (30 frames) in 89% of all tested sequences and within 0.5 seconds in 74% of sequences.
- Exact Hits: The model identified the exact ground-truth frame in 30.4% of cases.
- Robustness: The model successfully distinguished between complete and incomplete actions, achieving high accuracy even in challenging sports sequences with high motion blur.
Limitations and Future Work
- End-to-End Training: Due to hardware limitations, the authors used a two-stage approach (extracting CNN features first, then training the LSTM). They note that true end-to-end training remains a challenge for future work.
- Untrimmed Videos: The current model assumes the action is already temporally segmented (i.e., the video starts near the beginning of the attempt). Extending this to long, untrimmed videos is a primary goal for future research.
- Multiple Moments: The current formulation looks for a single completion moment. Future iterations could aim to detect multiple completion moments in complex, multi-step tasks.
Summary updated