Activity Auto-Completion: Predicting Human Activities from Partial Videos

Zhen Xu, Laiyun Qing, Jun MiaoICCV 2015 · DOI

Abstract

This paper introduces Activity Auto-Completion (AAC), a novel framework that treats human activity prediction as a ranking problem, analogous to search engine autocomplete. By representing partial videos as "prefixes" and full activities as "queries," the model provides a dynamically updated, ranked list of potential future actions. This approach addresses the inherent ambiguity in early-stage activity prediction. Using mid-level discriminative patches and a Learning to Rank formulation, AAC significantly outperforms traditional classification methods on standard benchmarks, achieving high accuracy even with only 50% of the video observed.

Introduction

Traditional computer vision systems excel at activity recognition—identifying an action after it has concluded. However, real-world applications such as autonomous surveillance, healthcare monitoring, and human-robot interaction require the ability to predict actions as they unfold. The primary challenge in early-stage prediction is ambiguity; for instance, an initial reaching motion could precede a handshake, a punch, or a simple gesture.

The concept of Activity Auto-completion, drawing an analogy between video prefixes and search engine queries.

Rather than forcing a single "hard" label prematurely, the authors propose an intelligent system that maintains a ranked list of possibilities, which refines as more temporal data arrives. This conceptual shift from classification to information retrieval (IR) forms the basis of the Activity Auto-Completion (AAC) framework.

Methodology

The AAC framework bridges the gap between IR and computer vision by formulating activity prediction as a Learning to Rank problem.

  1. Activity Representation: The model identifies "mid-level discriminative patches"—specific visual patterns like a raised arm or a particular stance—using a bank of SVMs. These patches serve as "visual characters."
  2. Video Segmentation: Videos are partitioned into variable-length segments called "activity characters," transforming a video into a sequence of symbols similar to characters in a word.
  3. Ranking Mechanism: Using the analogy of Query Auto-Completion (QAC), a partially observed video (prefix) is compared against a pool of full-video candidates (queries) from the training set. The system employs RankSVM to generate a relevance-based ranking, utilizing histogram intersection to measure the match between discriminative patches in the prefix and candidates.
  4. Refinement: A weighting scheme is implemented to boost the scores of detectors associated with the dominant activity class observed in the prefix, accounting for commonality and visual similarity.
The AAC framework pipeline, illustrating the transition from training sets to the Learning to Rank model and final testing.

Experimental Results

The model was evaluated on the UT-Interaction datasets (Set #1 and Set #2), which feature complex human interactions in both controlled and jittery environments.

  • Prediction Accuracy: On Set #1, AAC achieved 91.67% accuracy with only 50% of the video observed, significantly surpassing the previous state-of-the-art (MTSSVM at 78.33%). With full observation, accuracy reached 96.67%.
  • Ranking Quality: Mean Reciprocal Rank (MRR) analysis showed that even in very early stages (10-20% observation), the correct activity was typically within the top three candidates.

Discussion and Conclusion

The success of AAC highlights the importance of embracing uncertainty through ranking and the effectiveness of mid-level features in providing discriminative power across viewpoints. While certain activities like "pointing" are identifiable almost immediately, others like "punching" require more temporal context.

Future work may address the computational complexity of candidate lookup in massive datasets and explore more granular "activity characters" for fluid motions. The authors suggest that deeper exploration of the temporal evolution of mid-level features could improve the prediction of event timing.

Summary updated

BibTeX
@inproceedings{xu2015activity,
  author = {Zhen Xu and Laiyun Qing and Jun Miao},
  title = {{Activity Auto-Completion: Predicting Human Activities from Partial Videos}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.365},
}
Figure