Hierarchical Nonlinear Orthogonal Adaptive-Subspace Self-Organizing Map Based Feature Extraction for Human Action Recognition

Yang Du, Chunfeng Yuan, Bing Li, Weiming Hu, Hao Yang, Zhikang Fu, Lili ZhaoAAAI 2018 · DOI

Abstract

Human action recognition in video remains a significant challenge due to the complexity of spatio-temporal data and the high cost of manual labeling. This paper presents a novel unsupervised feature extraction method: the Nonlinear Orthogonal Adaptive-Subspace Self-Organizing Map (NOASSOM). By integrating a nonlinear orthogonal map layer and a hierarchical architecture, the proposed model effectively captures complex motion and appearance patterns without human supervision. Experimental results on benchmark datasets like UCF-101 and HMDB-51 demonstrate that NOASSOM outperforms several state-of-the-art supervised and hand-crafted methods.

Introduction

Feature extraction is a critical step in action recognition, traditionally divided between rigid hand-crafted descriptors (e.g., HOG, SIFT) and data-intensive supervised deep learning models (e.g., CNNs). While hand-crafted features often lack adaptability, deep learning models require massive labeled datasets to perform effectively. This research addresses the gap in unsupervised learning, proposing a system that can self-organize and discover meaningful patterns from raw video data without human-provided labels.

The NOASSOM Framework

The core innovation of this work is the evolution of the classical Adaptive-Subspace Self-Organizing Map (ASSOM) into a nonlinear domain. Standard ASSOM models are limited to linear projections, which are often insufficient for the high-dimensional, non-linear manifolds found in video data.

Comparison of the classical ASSOM (a) and the proposed NOASSOM (b) architectures, highlighting the addition of the nonlinear orthogonal map layer.

As illustrated in the architectural comparison, NOASSOM introduces a nonlinear orthogonal map layer utilizing a modified polynomial kernel trick. This layer projects input data into a higher-dimensional space where orthogonal subspaces can be more effectively learned. Furthermore, the researchers redesigned the loss function to support unsupervised learning from individual samples, allowing the network to "self-organize" based on inherent statistical patterns discovered in the data.

Hierarchical Architecture

To capture both appearance and motion information, the researchers implemented a multi-channel hierarchical structure. This approach mimics the deep stacking of neural networks while maintaining an unsupervised learning paradigm.

The overall hierarchical framework of the proposed system, illustrating the spatial, temporal, and fusion channels.

The system processes video through several distinct stages:

  1. Region Proposal: Improved Dense Trajectories (iDT) are used to identify relevant 3D volumes in the video where action is occurring.
  2. Spatial Channel: Learns appearance features from frame patches within the identified volumes.
  3. Temporal Channel: Learns motion features from sub-blocks of video volumes across continuous frames.
  4. Fusion Layer: A final third layer merges the spatial and temporal streams to create a comprehensive 400-dimensional spatio-temporal local feature descriptor.

Experimental Results

The NOASSOM model was evaluated against three major benchmarks: KTH, UCF-101, and HMDB-51. The results demonstrate that the nonlinear extension provides a significant performance boost (roughly 3.5% to 4.7%) over linear counterparts, confirming that raw video data is too complex for simple linear models.

  • HMDB-51: Achieved 69.3% accuracy, outperforming the Two-Stream CNN (59.4%) and TDD+iDT (65.9%).
  • UCF-101: Achieved 93.8% accuracy, surpassing many hybrid architectures and rivaling top-tier supervised models.
  • KTH: Reached a near-perfect 98.2% accuracy.

A key insight from the experiments was that the temporal channel (motion) is generally more discriminative for action recognition than the spatial channel (appearance), though fusing them consistently yields the best results.

Conclusion

NOASSOM represents a powerful advancement for unsupervised action recognition. By combining nonlinear kernel methods with a hierarchical self-organizing structure, it achieves results competitive with supervised deep learning while requiring significantly less labeled data and training time. Future work will explore deeper architectures and the application of NOASSOM as a generalized feature descriptor for other data modalities beyond video.

Summary updated

BibTeX
@inproceedings{du2018hierarchical,
  author = {Yang Du and Chunfeng Yuan and Bing Li and Weiming Hu and Hao Yang and Zhikang Fu and Lili Zhao},
  title = {{Hierarchical Nonlinear Orthogonal Adaptive-Subspace Self-Organizing Map Based Feature Extraction for Human Action Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12248},
}
Figure