Abstract
We propose two new models for human action recognition from video sequences using topic models. Video sequences are represented by a novel "bag-of-words" representation, where each frame corresponds to a "word". Our models differ from previous latent topic models for visual recognition in two major aspects: first of all, the latent topics in our models directly correspond to class labels; second, some of the latent variables in previous topic models become observed in our case. Our models have several advantages over other latent topic models used in visual recognition. First of all, the training is much easier due to the decoupling of the model parameters. Second, it alleviates the issue of how to choose the appropriate number of latent topics. Third, it achieves much better performance by utilizing the information provided by the class labels in the training set. We present action classification results on five different data sets. Our results are either comparable to, or significantly better than previously published results on these data sets.
Introduction
Recognizing human actions in video is a fundamental challenge in computer vision. Traditional approaches typically fall into two categories: complex dynamical models, such as Hidden Markov Models (HMMs) or Dynamic Bayesian Networks, and local "bag-of-words" models. While dynamical models are mathematically elegant, they are often brittle, difficult to train, and prone to overfitting due to a high number of parameters. Conversely, local bag-of-words models, which treat videos as collections of interest points (e.g., SIFT features), are robust but often fail to capture the global configuration and movement of the human body.
Standard Topic Models like Latent Dirichlet Allocation (LDA) have been applied to vision, but they suffer from a semantic gap: the "topics" discovered are purely mathematical clusters with no inherent link to action labels like "running" or "walking." This paper introduces Semi-Latent Topic Models (S-LDA and S-CTM) to bridge this gap by aligning latent topics directly with action labels during the training process.
Methodology: From Video to Words
The core of the proposed approach is a novel "bag-of-words" framework that treats each video frame as a single word. Instead of extracting thousands of tiny local patches, the system computes a large-scale motion descriptor for every frame using optical flow. This flow is split into horizontal and vertical components, rectified, and blurred to capture the overall shape and direction of motion.
These descriptors are clustered into a vocabulary of "motion words." A video sequence is then represented as a "document" containing a sequence of these words. In the Semi-Latent Topic Models (S-LDA and S-CTM), the topics are treated as observed variables during training. By enforcing a one-to-one correspondence between topics and action labels, the model parameters become decoupled. This decoupling allows for incredibly fast training—often under a minute—as it avoids the iterative, computationally expensive inference required by standard latent models.
Experimental Evaluation
The models were evaluated across five diverse and challenging datasets, demonstrating significant improvements over existing baselines:
- Weizmann: The model achieved a perfect 100% accuracy for per-video classification.
- KTH: The system reached 91.2% accuracy, significantly outperforming original SVM-based methods (71.72%) and contemporary interest-point methods.
- Soccer: On digitized World Cup footage, the semi-latent models outperformed a k-nearest neighbor (KNN) baseline by a massive margin (77.8% vs 49.2%).
- Ballet: The model achieved 91.36% accuracy, nearly doubling the performance of previous mid-level feature approaches (51%).
- Hockey: Evaluation on 70 tracks of players performing eight skating actions further confirmed the model's robustness.
Key Insights and Discussion
The success of S-LDA and S-CTM stems from three primary factors. First, direct supervision eliminates the "topic-to-label" mapping problem, forcing the model to learn the most discriminative motion words for each specific action. Second, the use of large-scale descriptors emphasizes the global configuration of the body, which proves more important for action recognition than local patches. Finally, while the bag-of-words approach technically ignores frame order, the topic model captures word co-occurrence. This provides an implicit temporal smoothing—the model "knows" certain motion words are likely to appear together in a "walking" sequence—offering a computationally efficient alternative to full temporal modeling.
Conclusion
Semi-Latent Topic Models provide a powerful and efficient framework for human action recognition. By aligning the mathematical structure of topic models with semantic class labels, the authors achieve state-of-the-art results while simplifying the training process. While the method currently depends on preprocessing for figure tracking and may lose fine-grained temporal ordering in some cases, its performance across diverse datasets establishes it as a robust solution for high-level motion analysis.
Summary updated