Eigen Evolution Pooling for Human Action Recognition

Yan Wang, Vinh Cao Trần, Minh HoaiarXiv 2017 · DOI · arXiv

Abstract

We introduce Eigen Evolution Pooling, an efficient method to aggregate a sequence of feature vectors. Eigen evolution pooling is designed to produce compact feature representations for a sequence of feature vectors, while maximally preserving as much information about the sequence as possible, especially the temporal evolution of the features over time. Eigen evolution pooling is a general pooling method that can be applied to any sequence of feature vectors, from low-level RGB values to high-level Convolutional Neural Network (CNN) feature vectors. We show that eigen evolution pooling is more effective than average, max, and rank pooling for encoding the dynamics of human actions in video. We demonstrate the power of eigen evolution pooling on UCF101 and Hollywood2 datasets, two human action recognition benchmarks, and achieve state-of-the-art performance.

Introduction

In computer vision, representing a video is significantly more complex than representing a static image. A video consists of a sequence of frames; to classify an action (e.g., "diving" vs. "jumping"), features from all frames must be aggregated—or "pooled"—into a single, fixed-length vector suitable for a classifier.

Standard industry approaches, such as average or max pooling, are "orderless," treating a video as a "bag of frames" and ignoring temporal sequence. While rank pooling attempts to capture trends, it often loses detailed fluctuations. Eigen Evolution Pooling (EEP) addresses this by compressing feature sequences into compact representations that preserve the temporal "story" of how features change over time.

Methodology

Eigen Evolution Pooling treats the evolution of each individual feature over time as a one-dimensional function. The process involves several key steps:

  1. Feature Decomposition: A sequence of $d$-dimensional feature vectors over $L$ time steps is viewed as $d$ separate temporal functions.
  2. Learning Eigen Evolutions: Principal Component Analysis (PCA) is applied to training videos to identify the "Principal Components" of these temporal functions, termed Eigen Evolution Functions.
  3. Projection: Feature sequences are projected onto the top $k$ eigen functions, resulting in $k$ compact descriptors ($EEP_1, EEP_2, \dots$) representing different "modes" of evolution.
  4. Eigen Images: When applied to raw RGB pixels, this process generates "Eigen Images"—visual summaries where the first image resembles a blurred average and subsequent images capture specific motion directions.
The Eigen Evolution Pooling pipeline: from feature sequences to compact EEP descriptors.

Key Insights: The DCT Connection

A significant technical discovery in this work is that learned data-driven eigen functions converge toward the basis functions of the Discrete Cosine Transform (DCT). This suggests that the "physics" of visual feature evolution in video follows a universal structure, allowing DCT to serve as a "plug-and-play" approximation that requires no training and handles variable video lengths.

Similarity between learned Eigen Evolution functions (RGB and TSN) and the Discrete Cosine Transform (DCT) basis.

Additional insights include:

  • Local Dynamics: Applying EEP within a sliding window ("local pooling") is more effective than global application, as it captures fine-grained movements.
  • Efficiency: As a linear operator, EEP is extremely fast, requiring only a single matrix-vector multiplication once basis functions are established.

Experimental Results

EEP was evaluated on two gold-standard benchmarks: UCF101 (13,320 clips, 101 categories) and Hollywood2 (1,707 clips, 12 action classes). The method consistently outperformed existing techniques:

  • Comparison to Rank Pooling: On UCF101 using RGB values, EEP achieved 72.7% accuracy, significantly higher than the 67.2% achieved by rank pooling (Dynamic Images).
  • State-of-the-Art Performance: Combined with Temporal Segment Networks (TSN), EEP reached 95.8% accuracy on UCF101 and 80.5% mAP on Hollywood2.
  • DCT Robustness: The DCT approximation yielded results nearly identical to the exact PCA-based method (e.g., 94.6% on UCF101 for deep features), proving its practical utility.

Conclusion

Eigen Evolution Pooling provides a powerful, modular way to aggregate temporal features. While currently relying on pre-extracted features, future work may involve integrating EEP directly into neural network architectures as an end-to-end trainable temporal pooling layer. Further exploration of signal-processing bases beyond DCT could also enhance the capture of sudden, non-periodic movements.

Summary updated

Abstract

We introduce Eigen Evolution Pooling, an efficient method to aggregate a sequence of feature vectors. Eigen evolution pooling is designed to produce compact feature representations for a sequence of feature vectors, while maximally preserving as much information about the sequence as possible, especially the temporal evolution of the features over time. Eigen evolution pooling is a general pooling method that can be applied to any sequence of feature vectors, from low-level RGB values to high-level Convolutional Neural Network (CNN) feature vectors. We show that eigen evolution pooling is more effective than average, max, and rank pooling for encoding the dynamics of human actions in video. We demonstrate the power of eigen evolution pooling on UCF101 and Hollywood2 datasets, two human action recognition benchmarks, and achieve state-of-the-art performance.
BibTeX
@misc{wang2017eigen,
  author = {Yan Wang and Vinh Cao Trần and Minh Hoai},
  title = {{Eigen Evolution Pooling for Human Action Recognition}},
  year = {2017},
  eprint = {1708.05465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.05465},
}
Figure