Introduction
Action recognition in video remains a formidable challenge in computer vision. Unlike static image classification, video analysis requires capturing complex temporal correlations across frames. While "two-stream" architectures—processing both spatial (RGB) and temporal (optical flow) information—have become standard, they often struggle with uncertainty quantification and horizontal scalability. Standard deep learning models typically act as "black boxes," providing predictions without a measure of confidence.
Pillar Networks++ addresses these issues by marrying the feature-extraction capabilities of deep convolutional neural networks (CNNs) with the probabilistic rigor of Gaussian Processes (GPs). By treating multiple deep networks as "experts" within a Product of Experts (PoE) framework, this approach achieves state-of-the-art performance while enabling robust uncertainty estimation.
Methodology: The Pillar Architecture
The core of the proposed framework is a modular "pillar" design. Each pillar consists of a deep neural network acting as a feature extractor, followed by a non-parametric Bayesian classifier.
The architecture utilizes four distinct deep streams:
- Inception-RGB and Inception-Flow
- ResNet-RGB and ResNet-Flow
Instead of standard softmax layers, features are fed into Gaussian Processes. To handle the $O(n^3)$ computational complexity of GPs, the training data is partitioned into seven subsets per stream. This results in 28 "experts" (4 streams × 7 subsets). These experts are then aggregated using a Product of Experts (PoE) formalism, where individual predictions are weighted by their respective certainties. This modularity allows for "horizontal scaling"—new architectures can be added as additional pillars without retraining the entire system.
Experimental Setup
The framework was evaluated on the HMDB51 dataset, a benchmark comprising 6,766 video clips across 51 action classes. HMDB51 is notably challenging due to real-world noise, including varying camera angles, occlusions, and diverse lighting conditions.
Key Results and Discussion
Pillar Networks++ demonstrates significant improvements over existing methods. The probabilistic fusion of multiple streams allows the model to bypass the need for manually engineered features, such as improved Dense Trajectories (iDT), which are often required by other high-performing models.
As shown in the results, the "Fusion-all" approach (combining all 28 GP experts) achieves a top accuracy of 73.6%, outperforming established benchmarks like the Temporal Segment Network (69.4%) and TS-LSTM (69%). Furthermore, the GP-PoE method consistently outperformed Multi-Kernel Learning (MKL) with Support Vector Machines (SVMs), validating the benefits of the Bayesian treatment of uncertainty in the fusion process.
Conclusion and Future Directions
Pillar Networks++ provides a scalable, probabilistic framework for video action recognition. By leveraging uncertainty as a feature during fusion, the model achieves superior accuracy and reliability.
Future work will focus on addressing the potential for overconfidence in Product of Experts models and exploring more sophisticated inference techniques, such as Variational Inference or Expectation Propagation. Additionally, the authors aim to test the horizontal scalability of the pillar architecture on massive datasets like Kinetics or YouTube-8m.
Summary updated