Combining Densely Sampled Form and Motion for Human Action Recognition

Konrad Schindler, Luc Van GoolDAGM-Symposium 2008 · DOI

TL;DR

This paper introduces a novel approach to human action recognition in videos by independently processing "form" (shape) and "motion" (optic flow) features. By combining these two biologically inspired channels, the method achieves state-of-the-art performance on standard datasets, demonstrating that a richer representation leads to more accurate action recognition.

What Problem Does This Paper Address?

Recognizing human actions in videos is a crucial task with applications ranging from surveillance to human-computer interaction. Previous methods often relied on a single type of feature, either focusing solely on motion (like optic flow) or attempting to capture both shape and motion within a single feature representation. This approach assumes that one feature type is sufficient or that a single feature can effectively represent both aspects. However, the authors argue that human visual systems process shape and motion separately, suggesting that a similar independent processing strategy might yield better results for artificial systems. The challenge is to effectively extract and combine these distinct cues to improve recognition accuracy.

Methods

The core of the proposed method lies in its two parallel processing streams: one for form and one for motion.

  • Form Channel: This channel extracts local shape information from each video frame independently. It uses a bank of log-Gabor filters, inspired by the human visual cortex, to detect oriented edges at multiple scales. To make the representation robust to translations, the filter responses are locally pooled using a MAX-operator. These pooled responses are then down-sampled and compared to a set of learned "templates" derived from principal component analysis (PCA) of the pooled orientation maps. This comparison yields a vector of similarity scores for the form channel.
  • Motion Channel: This channel focuses on local motion. It computes dense optic flow between consecutive frames. Unlike some methods, it does not smooth the flow field to preserve information at discontinuities. The optic flow is then processed similarly to the form channel: it's discretized into response maps for different flow directions and speeds, pooled using the MAX-operator, and compared to a set of learned motion templates (also derived via PCA). This results in a similarity score vector for the motion channel.

Finally, the similarity score vectors from both the form and motion channels are concatenated. This combined feature vector is then fed into a discriminative classifier (specifically, a bank of one-vs-all Support Vector Machines or SVMs) for the final action recognition. The authors note that a simple linear SVM is sufficient, and non-linear kernels do not provide significant improvement.

Datasets

The paper evaluates its method on two standard datasets for human action recognition:

  • WEIZMANN dataset: Consists of 9 subjects performing 9 different actions. For evaluation, sequences were trimmed to 28 frames, and 9-fold cross-validation was used.
  • KTH dataset: Consists of 25 subjects performing 6 different periodic actions. The dataset was recorded under four different conditions (outdoors, scale variations, different clothes, indoors). For evaluation, sequences were trimmed to 18 frames, and 5-fold cross-validation was used. The authors also mention mirroring sequences for data augmentation.

Key Insights and Ideas

  • Biologically Inspired Parallel Processing: The central idea of processing form and motion independently, mirroring the human visual system's ventral and dorsal pathways, is a key insight. This separation allows each channel to focus on its specific cue without interference.
  • Dense Feature Extraction: The use of densely sampled features (both shape and motion) rather than sparse interest points provides a richer representation of the action.
  • MAX-Pooling for Robustness: Employing MAX-pooling for local pooling in both channels enhances robustness to translations and preserves important feature responses.
  • PCA for Template Learning: Using PCA to learn templates for both form and motion features provides a compact and informative representation, effectively summarizing the visual information.
  • Form and Motion Synergy: The experiments clearly demonstrate that combining form and motion features consistently outperforms using either feature alone, validating the core hypothesis of the paper.

Key Results

  • State-of-the-Art Performance: The proposed method achieves perfect recognition (100%) on the WEIZMANN dataset and the best reported results to date on the KTH dataset.
  • WEIZMANN Dataset: Achieved 100% accuracy, which the authors note might indicate the dataset is too easy.
  • KTH Dataset: Achieved 92.7% accuracy (average across scenarios), outperforming previous methods like Niebles [20] (81.5%) and Dollar [8] (81.2%).
  • Optimal Weighting: The optimal relative weighting between form and motion features (λ) was found to be consistent across different datasets and robust to small variations, suggesting a general applicability of the combination strategy.
  • Confusion Analysis: Confusions on the KTH dataset primarily occurred between visually similar actions like "run-jog" and "walk-jog," which is intuitively understandable.

Limitations and Future Work

  • Invariance: The current system lacks explicit mechanisms for invariance against scale, rotation, and viewpoint changes. While it shows some robustness to scale and viewpoint variations on the KTH dataset, this is an area for future improvement.
  • Key Frames: An open question is identifying which frames within a motion sequence are most critical for recognition ("key-frames"), a topic relevant to both computer vision and neuroscience.
  • Training Cost: Training the system, particularly the template learning via Singular Value Decomposition, is computationally expensive. However, testing is efficient.
  • Optic Flow Computation: While testing is efficient, the optic flow computation itself can be a bottleneck, though it can be performed at frame-rate on modern GPUs.

Summary updated

BibTeX
@inproceedings{schindler2008combining,
  author = {Konrad Schindler and Luc Van Gool},
  title = {{Combining Densely Sampled Form and Motion for Human Action Recognition}},
  booktitle = {DAGM-Symposium},
  year = {2008},
  doi = {10.1007/978-3-540-69321-5\_13},
}
Figure