Pillar Networks for action recognition

Biswa Sengupta, Yu QianarXiv 2017 · arXiv

Introduction

Action recognition in video remains a significant challenge in computer vision due to the inherent temporal complexity and high computational costs associated with processing video data. Unlike static image classification, video understanding requires capturing patterns that evolve over time across varying camera angles, lighting conditions, and video qualities.

This paper introduces Pillar Networks, a modular framework designed to address these hurdles. By combining multiple deep learning architectures with classical hand-crafted features through Multi-Kernel Learning (MKL), the proposed method achieves state-of-the-art performance on the challenging HMDB-51 dataset.

The Pillar Architecture

The core innovation of this work is the "Pillar" metaphor: treating different feature extractors as parallel, independent streams that support a hierarchical classifier. This modularity allows for horizontal scalability; as new and more powerful neural networks are developed, they can be integrated into the framework as additional pillars without requiring a complete system redesign.

The Pillar Network architecture combines multiple deep learning streams (ResNet and Inception) for both RGB and motion data, alongside case-specific hand-crafted tensors.

Feature Extraction Streams

The framework utilizes a four-stream approach for deep feature extraction:

  • Deep CNNs: It employs two Inception networks and two Residual Networks (ResNets). For each architecture, one stream processes static RGB frames to capture spatial information, while the other processes optical flow to capture temporal motion.
  • Hand-crafted Features: To complement the deep networks, the authors include "case-specific tensors" such as Improved Dense Trajectories (iDT) and Multi-skip Feature Stacking (MIFS). these classical techniques help capture motion boundaries and frequency-domain information that standard CNNs might overlook.

Multi-Kernel Learning Fusion

To integrate these diverse information sources, the authors employ a Support Vector Machine (SVM) with Multi-Kernel Learning (MKL). Instead of simple feature concatenation, MKL learns the optimal weight for each feature source (kernel). This decoupled training approach allows the model to determine which pillars are most reliable for specific action recognition tasks, facilitating a "plug-and-play" flexibility for different network components.

Performance Benchmarks

The primary evaluation was conducted on the HMDB-51 dataset, which consists of 6,766 video clips across 51 action classes. HMDB-51 is widely regarded as more difficult than other datasets like UCF-101 due to its "in the wild" nature, featuring significant camera motion and occlusions.

Comparison of Pillar Networks against state-of-the-art action recognition methods on the HMDB-51 dataset.

As shown in the results, the Pillar Network approach consistently outperforms single-model baselines and previous state-of-the-art methods:

  • Inception Pillar alone: ~69.3% accuracy.
  • ResNet Pillar alone: ~63.2% accuracy.
  • Fused Pillar Networks (Deep only): 72.8% accuracy.
  • Full Pillar Networks (Deep + iDT + MIFS): 73.3% accuracy.

For comparison, the classic "Two-stream" approach achieves approximately 59.4%, while the popular TS-LSTM model reaches 69%. The Pillar Network's jump to 73.3% represents a substantial advancement for this benchmark.

Discussion and Future Work

The results demonstrate a clear "hybrid advantage." Even as deep learning matures, hand-crafted features like iDT still provide a marginal performance boost (~0.5%), suggesting they capture specific physical cues that deep networks are only beginning to learn implicitly. However, the authors acknowledge that as temporal modeling in CNNs improves, these classical methods may eventually become obsolete.

Future research directions include scaling this "wide and deep" architecture to massive datasets like YouTube-8M or Kinetics and exploring end-to-end training possibilities to further refine accuracy while maintaining the framework's modular benefits.

Summary updated

Abstract

Image understanding using deep convolutional network has reached human-level performance, yet a closely related problem of video understanding especially, action recognition has not reached the requisite level of maturity. We combine multi-kernels based support-vector-machines (SVM) with a multi-stream deep convolutional neural network to achieve close to state-of-the-art performance on a 51-class activity recognition problem (HMDB-51 dataset); this specific dataset has proved to be particularly challenging for deep neural networks due to the heterogeneity in camera viewpoints, video quality, etc. The resulting architecture is named pillar networks as each (very) deep neural network acts as a pillar for the hierarchical classifiers. In addition, we illustrate that hand-crafted features such as improved dense trajectories (iDT) and Multi-skip Feature Stacking (MIFS), as additional pillars, can further supplement the performance.
BibTeX
@misc{sengupta2017pillar,
  author = {Biswa Sengupta and Yu Qian},
  title = {{Pillar Networks for action recognition}},
  year = {2017},
  eprint = {1707.06923},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.06923},
}
Figure