Learning Mid-level Words on Riemannian Manifold for Action Recognition

Mengyi Liu, Ruiping Wang, Shiguang Shan, Xilin ChenarXiv 2015 · arXiv

Introduction

Human action recognition remains a significant challenge in computer vision due to the vast intra-class variations caused by changes in camera viewpoint, lighting conditions, and individual execution styles. Existing approaches typically rely on either low-level local features, which are robust but lack semantic depth, or high-level semantic models, which are descriptive but often fragile in unconstrained environments. This research proposes a middle ground: a mid-level representation that combines the robustness of local descriptors with the discriminative power of global statistical modeling.

The Proposed Framework

The core of this approach is a three-stage pipeline designed to bridge the gap between raw video data and high-level action categories. By modeling feature groups as points on curved Riemannian manifolds, the framework preserves the essential geometric structure of action dynamics.

Overview of the proposed framework for learning mid-level words on Riemannian manifolds.

1. Global Alignment via Universal GMM

The process begins with the extraction of dense low-level features such as HOG (Histograms of Oriented Gradients), HOF (Histograms of Optical Flow), and MBH (Motion Boundary Histograms). To ensure these features are comparable across different videos, a Universal Gaussian Mixture Model (GMM) is employed to align them into semantically related "feature groups." This alignment provides a common coordinate system, mitigating the impact of noise and background clutter.

2. Mid-level Word Modeling on Riemannian Manifolds

Rather than using simple vector averaging, the aligned feature groups are modeled using sophisticated statistics that reside on Riemannian manifolds:

  • Linear Subspaces: Represented on the Grassmann manifold.
  • Covariance Matrices: Represented on the Symmetric Positive Definite (SPD) manifold.
  • Gaussian Distributions: Embedded into SPD manifolds to capture both first and second-order statistics.

3. Riemannian Encoding

To create a final video descriptor, the authors extend traditional Euclidean encoding techniques to the Riemannian space. This includes Riemannian versions of Bag of Visual Words (BoVW) using K-Karcher-means, Vector of Locally Aggregated Descriptors (VLAD), and Fisher Vectors (FV).

Experimental Evaluation

The framework was rigorously tested on four major action recognition benchmarks: YouTube Action Database, UCF50, HMDB51, and the ASLAN (Action Similarity Labeling) dataset.

Example video pairs from the ASLAN dataset illustrating 'same' and 'not same' action categories.

The results demonstrate the effectiveness of the manifold-based approach:

  • YouTube: 90.3% accuracy.
  • UCF50: 90.7% accuracy.
  • HMDB51: 56.4% accuracy, a significant achievement given the complexity of this dataset.
  • Efficiency: The proposed two-stage encoding is substantially faster than previous mid-level methods like "Action Bank," reducing processing time from 1156s to 43.9s per video.

Key Insights and Future Directions

A primary takeaway is that geometry matters. Treating statistical descriptors as points on curved manifolds rather than flat Euclidean spaces preserves discriminative information that is otherwise lost. Furthermore, the fusion of different manifold representations (Subspace, Covariance, and Gaussian) provides a complementary view of action dynamics, leading to superior performance.

However, the method showed a performance dip when applied to MBH descriptors, likely because MBH encodes relative motion which may conflict with the absolute alignment of the Universal GMM. Future work will focus on developing Riemannian-based metric learning to further improve similarity tasks and extending the framework to temporal action localization.

Summary updated

Abstract

Human action recognition remains a challenging task due to the various sources of video data and large intra-class variations. It thus becomes one of the key issues in recent research to explore effective and robust representation to handle such challenges. In this paper, we propose a novel representation approach by constructing mid-level words in videos and encoding them on Riemannian manifold. Specifically, we first conduct a global alignment on the densely extracted low-level features to build a bank of corresponding feature groups, each of which can be statistically modeled as a mid-level word lying on some specific Riemannian manifold. Based on these mid-level words, we construct intrinsic Riemannian codebooks by employing K-Karcher-means clustering and Riemannian Gaussian Mixture Model, and consequently extend the Riemannian manifold version of three well studied encoding methods in Euclidean space, i.e. Bag of Visual Words (BoVW), Vector of Locally Aggregated Descriptors (VLAD), and Fisher Vector (FV), to obtain the final action video representations. Our method is evaluated in two tasks on four popular realistic datasets: action recognition on YouTube, UCF50, HMDB51 databases, and action similarity labeling on ASLAN database. In all cases, the reported results achieve very competitive performance with those most recent state-of-the-art works.
BibTeX
@misc{liu2015learning,
  author = {Mengyi Liu and Ruiping Wang and Shiguang Shan and Xilin Chen},
  title = {{Learning Mid-level Words on Riemannian Manifold for Action Recognition}},
  year = {2015},
  eprint = {1511.04808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.04808},
}
Figure