Deep Architectures and Ensembles for Semantic Video Classification

Eng-Jon Ong, Sameed Husain, Mikel Bober-Irizar, Miroslaw Bober

Abstract

This paper addresses the challenge of large-scale semantic video classification by introducing a lightweight Fully Connected ResNet (FCRN) architecture and four novel ensembling strategies. Evaluated on the massive YouTube-8M dataset, the proposed methods achieve a state-of-the-art Global Average Precision (GAP20) of 85.12%. The findings demonstrate that simple video-level statistical features, when processed through optimized architectures and ensembled effectively, can match or exceed the performance of more complex temporal models.

Introduction

Accurate video classification is a complex task requiring the integration of spatial (visual) and temporal (audio/motion) cues. While Deep Neural Networks (DNNs) have revolutionized image classification, their application to video often results in prohibitive computational complexity. This research explores whether it is more effective to train single massive end-to-end networks or to ensemble diverse expert models, and identifies the mathematically optimal ways to combine these models without overfitting.

Architectural Diversity in Video Classification

The authors evaluate a wide array of architectures to determine the best performers for large-scale tasks. These include recurrent networks (LSTMs, GRUs) and temporal-agnostic models that utilize feature pooling (NetVLAD, NetFV, NetBOW).

Overview of the diverse neural network architectures evaluated, including recurrent, pooling-based, and residual models.

A primary finding is that diversity is critical; while individual models often hit a performance ceiling around 82-83% GAP20, they tend to excel on different semantic classes. This error profile diversity makes them ideal candidates for ensembling.

Fully Connected ResNet (FCRN)

A major contribution of this work is the Fully Connected ResNet (FCRN). Rather than processing raw video frames, the FCRN operates on highly compressed 2304-dimensional vectors representing the mean and standard deviation of frame-level features extracted via Inception networks.

The Residual Network block (left) and the overall ResNet pipeline (right) for video-level feature transformation.

This approach drastically reduces computational overhead. Surprisingly, the FCRN performed as well as complex temporal models like LSTMs, suggesting that for many semantic labels, global statistical summaries are sufficient.

Novel Ensembling Strategies

To push past individual model plateaus, the authors proposed four ensembling methods:

  • Correlation-based Ensembling: A greedy algorithm that pairs models with low Pearson correlation and optimizes weights via quadratic curve fitting.
  • Single Coefficient MoE: A Mixture of Experts (MoE) approach using a 1x1 convolutional layer to learn one optimal weight per DNN.
  • Per-class Coefficient MoE: Learns separate weights for every class for every DNN, though this was found to be highly prone to overfitting.
  • Regularized Dual-Stream MoE: A hybrid that learns a base weight per DNN and a regularized residual weight per class to balance flexibility and generalization.

Key Results and Insights

  • State-of-the-Art Performance: The Single Coefficient MoE ensemble of 13 diverse models achieved a 85.12% GAP20 on the YouTube-8M Kaggle test set, surpassing previous brute-force averaging methods.
  • Model Contributions: Interestingly, the ensemble learning algorithms assigned negative weights to LSTM models, indicating they provided redundant or less useful information compared to the FCRN and pooling-based models.
  • Transfer Learning: The ensemble demonstrated strong generalization, achieving 84.06% on UCF101 and 52.61% on HMDB51, competitive with methods using much heavier full-frame RGB features.

Conclusion

The study concludes that simple statistical features are remarkably effective for large-scale video classification when paired with residual architectures. While complex ensembling can lead to overfitting, simpler MoE strategies successfully leverage model diversity to achieve state-of-the-art results.

Summary updated

BibTeX
@article{ong2019deep,
  author = {Eng-Jon Ong and Sameed Husain and Mikel Bober-Irizar and Miroslaw Bober},
  title = {{Deep Architectures and Ensembles for Semantic Video Classification}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2019},
  url = {https://openreview.net/forum?id=EAvGh0pJwm},
}
Figure