Relevance Topic Model for Unstructured Social Group Activity Recognition

Fang Zhao, Yongzhen Huang, Liang Wang, Tieniu Tan

This paper introduces the Relevance Topic Model (RTM), a novel framework designed to recognize complex social activities in web videos—even when labeled training data is extremely scarce. By combining undirected topic modeling with sparse Bayesian learning, RTM automatically identifies the most "relevant" features for classification, significantly outperforming existing methods in low-data scenarios.

The Challenge of Unstructured Social Activities

Classifying simple actions like "walking" or "clapping" in controlled environments is a largely solved problem. However, recognizing unstructured social group activities in web videos—such as a wedding dance, a birthday party, or a graduation ceremony—is significantly harder.

Examples of unstructured social group activities

The authors identify two primary hurdles in this domain:

  1. The Semantic Gap: There is a massive disconnect between low-level visual features (like pixels or gradients) and high-level semantic concepts (like "graduation ceremony").
  2. Data Scarcity: While web videos are plentiful, labeled training data for complex social events is often limited and expensive to acquire.

Traditional supervised topic models often struggle to generalize or require manual human annotations of "attributes" (e.g., "is there a cake?") to function well. The RTM aims to discover these discriminative mid-level representations automatically.

The Relevance Topic Model (RTM)

The RTM is a "mixed" graphical model that builds upon the Replicated Softmax—an undirected topic model often used for text. The authors introduce several key innovations to adapt this for video recognition:

  • Rectified Linear Units (ReLU) for Topics: Instead of using traditional binary stochastic units to represent topics, the RTM uses ReLUs. This allows the model to capture the relative importance of different topics and makes variational inference more tractable.
  • Sparse Bayesian Learning (SBL): The model incorporates an "Automatic Relevance Determination" (ARD) prior. This effectively "switches off" topics that aren't useful for classification, forcing the model to focus only on "relevance topics."
  • Joint Learning: Unlike pipeline methods that extract features and then train a classifier independently, RTM jointly learns the latent topics and the classifier weights.
  • Variational EM Algorithm: To handle complex probability distributions, the authors developed an efficient inference algorithm using a quadratic bound on the log-sum-exp function.

Datasets and Evaluation

The model was evaluated on the Unstructured Social Activity Attribute (USAA) dataset, which contains 8 semantic classes (e.g., music performance, parade, wedding reception) across 1,466 videos.

The authors used a "Bag-of-Words" (BoW) approach combining three modalities:

  • SIFT (Static appearance)
  • STIP (Motion/Spatio-temporal interest points)
  • MFCC (Auditory/Audio features)

Key Insights

  • Sparsity as a Shield Against Overfitting: By using sparse Bayesian learning, the model inherently resists overfitting. This is why it performs so well when training samples are limited; it refuses to "memorize" noise and instead hunts for the most robust topical signals.
  • Undirected is Better for Counts: Most topic models (like LDA) are directed. By using an undirected Replicated Softmax base, the RTM better models the distribution of word counts in a video, which is often sparse and high-dimensional.
  • Information Preservation: Moving from binary units to ReLUs isn't just a computational trick; it allows the model to express more nuanced information about the video content, which is necessary for complex scenes containing multiple objects and interactions.

Performance Results

The RTM was compared against other supervised topic models like MedLDA and gClassRBM.

  • Single-Modal Superiority: RTM consistently achieved higher accuracy across all feature types. For example, with 40 topics on SIFT features, RTM hit 55.83% accuracy compared to MedLDA’s 43.07%.
  • The "Small Data" Win: The most striking result occurred when training data was limited to just 10 instances per class. In this scenario, Multimodal RTM achieved ~45% accuracy, vastly outperforming a standard SVM (29%) and other state-of-the-art attribute-based models (40%).
  • Interpretability: The authors demonstrated that the learned topics are highly sparse, meaning the model successfully identifies specific, relevant clusters of features for each activity class.

Limitations and Future Work

While the RTM shows strong performance, the paper leaves a few areas for further exploration:

  • Computational Complexity: While the variational EM algorithm is described as efficient, a detailed comparison of training times against simpler discriminative models is needed.
  • Temporal Dynamics: The model uses a Bag-of-Words representation, which ignores the order of events in a video. Integrating temporal modeling (like HMMs or RNNs) could further improve recognition.
  • Scalability: Testing the model on "in-the-wild" datasets with hundreds of activity classes would be a logical next step to prove its robustness at scale.

Summary updated

Abstract

Unstructured social group activity recognition in web videos is a challenging task due to 1) the semantic gap between class labels and low-level visual features and 2) the lack of labeled training data. To tackle this problem, we propose a relevance topic model" for jointly learning meaningful mid-level representations upon bag-of-words (BoW) video representations and a classifier with sparse weights. In our approach, sparse Bayesian learning is incorporated into an undirected topic model (i.e., Replicated Softmax) to discover topics which are relevant to video classes and suitable for prediction. Rectified linear units are utilized to increase the expressive power of topics so as to explain better video data containing complex contents and make variational inference tractable for the proposed model. An efficient variational EM algorithm is presented for model parameter estimation and inference. Experimental results on the Unstructured Social Activity Attribute dataset show that our model achieves state of the art performance and outperforms other supervised topic model in terms of classification accuracy, particularly in the case of a very small number of labeled training videos."
BibTeX
@inproceedings{zhao2013relevance,
  author = {Fang Zhao and Yongzhen Huang and Liang Wang and Tieniu Tan},
  title = {{Relevance Topic Model for Unstructured Social Group Activity Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
}
Figure