Abstract
This paper introduces Activity Auto-Completion (AAC), a novel framework that treats human activity prediction as a ranking problem, analogous to search engine autocomplete. By representing partial videos as "prefixes" and full activities as "queries," the model provides a dynamically updated, ranked list of potential future actions. This approach addresses the inherent ambiguity in early-stage activity prediction. Using mid-level discriminative patches and a Learning to Rank formulation, AAC significantly outperforms traditional classification methods on standard benchmarks, achieving high accuracy even with only 50% of the video observed.
Introduction
Traditional computer vision systems excel at activity recognition—identifying an action after it has concluded. However, real-world applications such as autonomous surveillance, healthcare monitoring, and human-robot interaction require the ability to predict actions as they unfold. The primary challenge in early-stage prediction is ambiguity; for instance, an initial reaching motion could precede a handshake, a punch, or a simple gesture.
Rather than forcing a single "hard" label prematurely, the authors propose an intelligent system that maintains a ranked list of possibilities, which refines as more temporal data arrives. This conceptual shift from classification to information retrieval (IR) forms the basis of the Activity Auto-Completion (AAC) framework.
Methodology
The AAC framework bridges the gap between IR and computer vision by formulating activity prediction as a Learning to Rank problem.
- Activity Representation: The model identifies "mid-level discriminative patches"—specific visual patterns like a raised arm or a particular stance—using a bank of SVMs. These patches serve as "visual characters."
- Video Segmentation: Videos are partitioned into variable-length segments called "activity characters," transforming a video into a sequence of symbols similar to characters in a word.
- Ranking Mechanism: Using the analogy of Query Auto-Completion (QAC), a partially observed video (prefix) is compared against a pool of full-video candidates (queries) from the training set. The system employs RankSVM to generate a relevance-based ranking, utilizing histogram intersection to measure the match between discriminative patches in the prefix and candidates.
- Refinement: A weighting scheme is implemented to boost the scores of detectors associated with the dominant activity class observed in the prefix, accounting for commonality and visual similarity.
Experimental Results
The model was evaluated on the UT-Interaction datasets (Set #1 and Set #2), which feature complex human interactions in both controlled and jittery environments.
- Prediction Accuracy: On Set #1, AAC achieved 91.67% accuracy with only 50% of the video observed, significantly surpassing the previous state-of-the-art (MTSSVM at 78.33%). With full observation, accuracy reached 96.67%.
- Ranking Quality: Mean Reciprocal Rank (MRR) analysis showed that even in very early stages (10-20% observation), the correct activity was typically within the top three candidates.
Discussion and Conclusion
The success of AAC highlights the importance of embracing uncertainty through ranking and the effectiveness of mid-level features in providing discriminative power across viewpoints. While certain activities like "pointing" are identifiable almost immediately, others like "punching" require more temporal context.
Future work may address the computational complexity of candidate lookup in massive datasets and explore more granular "activity characters" for fluid motions. The authors suggest that deeper exploration of the temporal evolution of mid-level features could improve the prediction of event timing.
Summary updated