Attribute Learning for Understanding Unstructured Social Activity

Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Shaogang Gong

Understanding complex social activities in consumer videos is a significant challenge for computer vision due to the unstructured nature of the content and the high levels of noise. This paper introduces a "semi-latent" attribute learning framework designed to bridge the gap between human-defined concepts and the messy reality of real-world social media clips.

The Problem: Unstructured Social Video

Most research on visual attributes focuses on clean, single-object images or staged action clips. In contrast, social videos uploaded to platforms like YouTube are much more complex. They feature multiple people, varying audio, and "unstructured" contexts—for instance, a "birthday party" video might focus on a conversation rather than a cake.

The authors identify three primary gaps in existing models: incompleteness (humans cannot label every discriminative feature), subjectivity (manual annotations are noisy), and complexity (social activities are multi-modal and contain significant background noise).

The Semi-Latent Attribute Space (SLAS) Model

To address these issues, the authors propose the Semi-Latent Attribute Space (SLAS) model. Instead of relying solely on a fixed list of human-provided attributes, they treat attribute learning as a probabilistic topic modeling problem, generalizing Latent Dirichlet Allocation (LDA).

Overview of the SLAS framework and the semi-latent attribute space.

The SLAS model partitions the attribute space into three distinct types of topics:

  • User-Defined (UD) Attributes: These are the 69 manual labels (e.g., "singing," "candles"). The model is constrained to follow these labels when available.
  • Class-Conditional (CC) Latent Attributes: These are discovered automatically and forced to be discriminative for specific classes, capturing visual or audio patterns that humans might miss.
  • Background Non-discriminative (BN) Latent Attributes: These are learned in an unsupervised way to "soak up" noise common in consumer videos, such as random camera movement or background chatter, preventing this noise from polluting the classification.

For Zero-Shot Learning (ZSL), the framework uses a self-training algorithm. It leverages known UD attributes to find confident matches in unlabeled test data, which then helps estimate the latent attributes for new, unseen classes.

The USAA Dataset

The paper introduces the Unstructured Social Activity Attribute (USAA) dataset to benchmark this approach. Built on top of the Columbia Consumer Video (CCV) dataset, it includes approximately 1,500 videos across 8 classes, such as Wedding Ceremony, Graduation Party, and Parade.

Multi-modal attribute annotations for unstructured social activity videos.

As illustrated above, the dataset features 69 multi-modal binary attributes covering actions, objects, scenes, sounds, and camera movements. The model utilizes a combination of visual (SIFT), spatio-temporal (STIP), and audio (MFCC) features to represent these complex activities.

Key Insights and Results

The SLAS model consistently outperformed standard Direct Attribute Prediction (DAP) and raw classifiers (SVM/KNN), particularly in data-sparse scenarios:

  • Sparse Data Performance: With only 10 training instances and a tiny subset of 7 attributes, SLAS achieved 36% accuracy, compared to 27% for standard attribute models.
  • Zero-Shot Learning: In recognizing classes never seen during training, SLAS reached 45% accuracy, a significant jump over the 38% achieved by the then-standard SVM-DAP approach.
  • Noise Robustness: When 25% of the attribute labels were intentionally corrupted, the SLAS model's performance remained virtually unchanged, demonstrating the effectiveness of its semantic denoising.

Conclusion and Future Work

The SLAS framework demonstrates that "latent" attributes can often be more effective than exhaustive manual labeling, as computable discriminative features do not always align with nameable human concepts. Future work will focus on modeling attribute correlations (e.g., the relationship between "bride" and "wedding dress") and improving the fusion of different modalities like audio and video.

Summary updated

Abstract

The rapid development of social video sharing platforms has created a huge demand for automatic video classification and annotation techniques, in particular for videos containing social activities of a group of people (e.g. YouTube video of a wedding reception). Recently, attribute learning has emerged as a promising paradigm for transferring learning to sparsely labelled classes in object or single-object short action classification. In contrast to existing work, this paper for the first time, tackles the problem of attribute learning for understanding group social activities with sparse labels. This problem is more challenging because of the complex multi-object nature of social activities, and the unstructured nature of the activity context. To solve this problem, we (1) contribute an unstructured social activity attribute (USAA) dataset with both visual and audio attributes, (2) introduce the concept of semi-latent attribute space and (3) propose a novel model for learning the latent attributes which alleviate the dependence of existing models on exact and exhaustive manual specification of the attribute-space. We show that our framework is able to exploit latent attributes to outperform contemporary approaches for addressing a variety of realistic multi-media sparse data learning tasks including: multi-task learning, N-shot transfer learning, learning with label noise and importantly zero-shot learning.
BibTeX
@inproceedings{fu2012attribute,
  author = {Yanwei Fu and Timothy M. Hospedales and Tao Xiang and Shaogang Gong},
  title = {{Attribute Learning for Understanding Unstructured Social Activity}},
  booktitle = {European Conference on Computer Vision},
  year = {2012},
  doi = {10.1007/978-3-642-33765-9\_38},
}
Figure