Spatio-temporal Aware Non-negative Component Representation for Action Recognition

Jianhong Wang, Tian Lan, Xu Zhang, Limin Luo

Abstract

This paper presents a novel mid-level representation for action recognition, named spatio-temporal aware non-negative component representation (STANNCR). The proposed STANNCR is based on action components and incorporates spatial-temporal information. We first introduce a spatial-temporal distribution vector (STDV) to model the distributions of local feature locations in a compact and discriminative manner. Then we employ non-negative matrix factorization (NMF) to learn the action components and encode the video samples. The action component considers the correlations of visual words, which effectively bridge the semantic gap in action recognition. To incorporate the spatial-temporal cues for final representation, the STDV is used as part of graph regularization for NMF. The fusion of spatial-temporal information makes the STANNCR more discriminative, and our fusion manner is more compact than traditional methods of concatenating vectors. The proposed approach is extensively evaluated on three public datasets, demonstrating the effectiveness of STANNCR for action recognition.

Introduction

Action recognition in video remains a challenging task due to the "semantic gap" between low-level visual features and high-level human actions. The traditional Bag of Visual Words (BoVW) approach, while popular, suffers from two primary limitations: the loss of contextual relationships between visual words and a lack of spatial-temporal awareness. While methods like Spatial-Temporal Pyramids (STP) attempt to address the latter, they often result in high-dimensional feature vectors that are computationally burdensome.

This paper introduces STANNCR, a mid-level representation designed to bridge this gap. By grouping visual words into semantically meaningful "action components" and integrating spatial-temporal layouts via a novel distribution vector, STANNCR provides a compact yet highly discriminative feature set for action classification.

Methodology

The STANNCR framework consists of two main stages: modeling the spatial-temporal distribution of features and learning the mid-level action components through regularized matrix factorization.

Overview of the STANNCR training and testing pipeline.

Spatio-temporal Distribution Vector (STDV)

To capture the "where" and "when" of an action without the dimensionality explosion of grid-based methods, the authors propose the STDV. This vector models the distribution of feature locations using a Gaussian Mixture Model (GMM) and Fisher Vectors. This approach provides a compact summary of the action's layout, ensuring that spatial and temporal cues are preserved in a low-dimensional format.

Action Components and ST-GNMF

The core of the representation is learned using Spatio-temporal Aware Graph Regularized Non-negative Matrix Factorization (ST-GNMF).

Visual words are decomposed into mid-level action components (e.g., $u_1, u_2$) to bridge the semantic gap.

As shown in the figure above, NMF decomposes video data into "action components"—mid-level units composed of correlated visual words (e.g., a specific limb movement). The STDV is integrated as a graph regularization constraint, ensuring that videos with similar spatial-temporal layouts are mapped to similar mid-level representations. The non-negativity constraint ensures that the final representation is purely additive, mirroring the intuitive composition of actions from distinct parts.

Experimental Evaluation

The proposed method was evaluated on three benchmark datasets: KTH, YouTube, and HMDB51.

  • KTH: A controlled dataset with 6 actions. STANNCR achieved a state-of-the-art accuracy of 97.8%.
  • YouTube: A challenging dataset with 11 categories and significant camera motion. The model reached 89.2% accuracy.
  • HMDB51: A large-scale dataset with 51 categories. STANNCR achieved 55.4% accuracy, outperforming existing benchmarks.

The results consistently showed that STANNCR outperforms BoVW and standard GNMF across various feature types, including Histogram of Oriented Gradients (HOG), Histogram of Optical Flow (HOF), and Motion Boundary Histograms (MBH). Furthermore, the STDV was shown to be more efficient and accurate than traditional Spatial-Temporal Pyramids.

Conclusion

STANNCR effectively addresses the limitations of traditional action recognition models by introducing a mid-level representation that is both semantically rich and spatially aware. By utilizing STDV for compact spatial-temporal modeling and ST-GNMF for learning action components, the framework achieves superior performance on diverse datasets while maintaining computational efficiency. Future work may explore deeper semantic hierarchies or the integration of these components into deep learning architectures.

Summary updated

Abstract

This paper presents a novel mid-level representation for action recognition, named spatio-temporal aware non-negative component representation (STANNCR). The proposed STANNCR is based on action component and incorporates the spatial-temporal information. We first introduce a spatial-temporal distribution vector (STDV) to model the distributions of local feature locations in a compact and discriminative manner. Then we employ non-negative matrix factorization (NMF) to learn the action components and encode the video samples. The action component considers the correlations of visual words, which effectively bridge the sematic gap in action recognition. To incorporate the spatial-temporal cues for final representation, the STDV is used as the part of graph regularization for NMF. The fusion of spatial-temporal information makes the STANNCR more discriminative, and our fusion manner is more compact than traditional method of concatenating vectors. The proposed approach is extensively evaluated on three public datasets. The experimental results demonstrate the effectiveness of STANNCR for action recognition.
BibTeX
@misc{wang2016spatio,
  author = {Jianhong Wang and Tian Lan and Xu Zhang and Limin Luo},
  title = {{Spatio-temporal Aware Non-negative Component Representation for Action Recognition}},
  year = {2016},
  eprint = {1608.07664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1608.07664},
}
Figure