Action Recognition Using Local Consistent Group Sparse Coding with Spatio-Temporal Structure

Yi Tian, Qiuqi Ruan, Gaoyun An, Yun FuACM Multimedia 2016 · DOI

TL;DR

This paper introduces a novel framework for human action recognition that effectively combines local feature representations with spatio-temporal structures. By proposing a "Local Consistent Group Sparse Coding" (LCGSC) method and an "Average Location" (AL) model, the authors achieve state-of-the-art performance on challenging datasets while maintaining computational efficiency.

What Problem Does This Paper Address?

Recognizing human actions in videos is a crucial task in computer vision with applications ranging from human-computer interaction to surveillance. While recent advancements have leveraged local spatio-temporal features, effectively integrating these features into a discriminative representation remains a challenge. Existing methods often overlook important information such as global feature structures, local correlations between features, and the spatio-temporal layout of actions. This can lead to representations that are sensitive to feature variations, lose crucial contextual information, or are computationally expensive. The paper aims to address these limitations by proposing a more comprehensive approach to action recognition.

Methods

The proposed framework consists of two main novel components:

  • Local Consistent Group Sparse Coding (LCGSC): This method extends traditional sparse coding by treating all local features within a video as a single group. It incorporates a "group sparsity" term to capture global structural information, ensuring that if a visual word is used to describe one feature, it's also considered for other features in the group. Crucially, it adds a "local consistent" regularization term to preserve local correlations between similar features. This dual approach aims to create more informative and discriminative representations compared to standard group sparse coding.
  • Average Location (AL) Model: After obtaining sparse codes, this model captures the spatio-temporal structure of actions. It calculates the global mean and variance of visual word distributions across spatio-temporal coordinates. This is achieved by estimating the probability of each feature belonging to a given visual word, considering sparse codes, contextual information, and feature-word correlations. The AL model then uses these probabilities and the actual feature locations to generate a compact spatio-temporal descriptor for the video.

The final action representation is a concatenation of the LCGSC representation and the AL model's spatio-temporal descriptor, balanced by a learned factor.

Datasets

The proposed framework was evaluated on three standard datasets:

  • UCF sports dataset: A challenging dataset for action recognition.
  • Olympic sports dataset: A dataset noted for its real-world complexity.
  • YouTube dataset: A large-scale dataset of user-uploaded videos.

The paper mentions using a leave-one-out cross-validation strategy for the UCF and YouTube datasets, and a specific split for the Olympic dataset (134 videos for testing).

Key Insights and Ideas

  • Group sparsity is good, but local consistency is key: While group sparsity helps capture global structure, the paper highlights that neglecting local correlations between features can lead to confusions. LCGSC's addition of local consistency significantly improves discriminative power.
  • Spatio-temporal layout matters: The AL model demonstrates that explicitly modeling the spatial and temporal distribution of visual words provides crucial complementary information to appearance and motion cues.
  • Compact and efficient representations: The proposed LCGSC-AL representation is shown to be compact (e.g., 7|D| dimensions for the AL part, where |D| is the dictionary size) and computationally efficient, outperforming other spatio-temporal methods in terms of both accuracy and representation dimensionality.
  • Beyond simple pooling: Replacing traditional pooling operations with p-norm calculations on sparse codes strengthens feature characteristics and makes the representation less dependent on the number of features.

Key Results

  • State-of-the-art performance: The LCGSC-AL framework achieved the highest accuracies on the Olympic sports dataset (80.0%) and YouTube dataset (87.1%), and comparable performance on the UCF sports dataset (90.0%) compared to other significant methods.
  • LCGSC superiority: LCGSC alone outperformed standard sparse coding (SC), LLC, and GSC by significant margins (e.g., 4.7%, 1.5%, and 1.3% improvements on the three datasets, respectively).
  • AL model effectiveness: When combined with LCGSC, the AL model consistently and significantly improved performance over LCGSC alone, demonstrating its value in capturing spatio-temporal information.
  • Computational efficiency: The paper notes that LCGSC encoding is only slightly more computationally intensive than GSC, and the AL model provides compact representations that contribute to fast training speeds.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations acknowledged by the authors. However, the reliance on detected local features means the performance is inherently tied to the quality of this initial feature extraction. The choice of dictionary size and regularization parameters (λ1, λ2, β, α, γ) might require careful tuning.
  • Future Work: Not explicitly discussed in the paper. However, potential future directions could involve exploring more advanced feature detection methods, investigating adaptive dictionary learning, or applying the framework to other video analysis tasks.

Summary updated

BibTeX
@inproceedings{tian2016action,
  author = {Yi Tian and Qiuqi Ruan and Gaoyun An and Yun Fu},
  title = {{Action Recognition Using Local Consistent Group Sparse Coding with Spatio-Temporal Structure}},
  booktitle = {ACM Multimedia},
  year = {2016},
  doi = {10.1145/2964284.2967234},
}

Loading related content...

Figure