Pillar Networks++: Distributed non-parametric deep and wide networks

Biswa Sengupta, Yu QianarXiv 2017 · arXiv

Introduction

Action recognition in video remains a formidable challenge in computer vision. Unlike static image classification, video analysis requires capturing complex temporal correlations across frames. While "two-stream" architectures—processing both spatial (RGB) and temporal (optical flow) information—have become standard, they often struggle with uncertainty quantification and horizontal scalability. Standard deep learning models typically act as "black boxes," providing predictions without a measure of confidence.

Pillar Networks++ addresses these issues by marrying the feature-extraction capabilities of deep convolutional neural networks (CNNs) with the probabilistic rigor of Gaussian Processes (GPs). By treating multiple deep networks as "experts" within a Product of Experts (PoE) framework, this approach achieves state-of-the-art performance while enabling robust uncertainty estimation.

Methodology: The Pillar Architecture

The core of the proposed framework is a modular "pillar" design. Each pillar consists of a deep neural network acting as a feature extractor, followed by a non-parametric Bayesian classifier.

Overview of the Pillar Networks++ architecture, illustrating the fusion of multiple deep learning streams (ResNet and Inception) with Gaussian Process experts.

The architecture utilizes four distinct deep streams:

  1. Inception-RGB and Inception-Flow
  2. ResNet-RGB and ResNet-Flow

Instead of standard softmax layers, features are fed into Gaussian Processes. To handle the $O(n^3)$ computational complexity of GPs, the training data is partitioned into seven subsets per stream. This results in 28 "experts" (4 streams × 7 subsets). These experts are then aggregated using a Product of Experts (PoE) formalism, where individual predictions are weighted by their respective certainties. This modularity allows for "horizontal scaling"—new architectures can be added as additional pillars without retraining the entire system.

Experimental Setup

The framework was evaluated on the HMDB51 dataset, a benchmark comprising 6,766 video clips across 51 action classes. HMDB51 is notably challenging due to real-world noise, including varying camera angles, occlusions, and diverse lighting conditions.

Key Results and Discussion

Pillar Networks++ demonstrates significant improvements over existing methods. The probabilistic fusion of multiple streams allows the model to bypass the need for manually engineered features, such as improved Dense Trajectories (iDT), which are often required by other high-performing models.

Comparison of Pillar Networks++ accuracy against state-of-the-art methods on the HMDB51 dataset.

As shown in the results, the "Fusion-all" approach (combining all 28 GP experts) achieves a top accuracy of 73.6%, outperforming established benchmarks like the Temporal Segment Network (69.4%) and TS-LSTM (69%). Furthermore, the GP-PoE method consistently outperformed Multi-Kernel Learning (MKL) with Support Vector Machines (SVMs), validating the benefits of the Bayesian treatment of uncertainty in the fusion process.

Conclusion and Future Directions

Pillar Networks++ provides a scalable, probabilistic framework for video action recognition. By leveraging uncertainty as a feature during fusion, the model achieves superior accuracy and reliability.

Future work will focus on addressing the potential for overconfidence in Product of Experts models and exploring more sophisticated inference techniques, such as Variational Inference or Expectation Propagation. Additionally, the authors aim to test the horizontal scalability of the pillar architecture on massive datasets like Kinetics or YouTube-8m.

Summary updated

Abstract

In recent work, it was shown that combining multi-kernel based support vector machines (SVMs) can lead to near state-of-the-art performance on an action recognition dataset (HMDB-51 dataset). This was 0.4\% lower than frameworks that used hand-crafted features in addition to the deep convolutional feature extractors. In the present work, we show that combining distributed Gaussian Processes with multi-stream deep convolutional neural networks (CNN) alleviate the need to augment a neural network with hand-crafted features. In contrast to prior work, we treat each deep neural convolutional network as an expert wherein the individual predictions (and their respective uncertainties) are combined into a Product of Experts (PoE) framework.
BibTeX
@misc{sengupta2017pillar,
  author = {Biswa Sengupta and Yu Qian},
  title = {{Pillar Networks++: Distributed non-parametric deep and wide networks}},
  year = {2017},
  eprint = {1708.06250},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.06250},
}
Figure