Discriminative Video Representation Learning Using Support Vector Classifiers

Jue Wang, Anoop Cherian

Abstract

Standard video action recognition models often rely on simple average or max-pooling to aggregate frame-level features. However, these methods frequently fail to distinguish between relevant action frames and background noise, leading to diluted representations. This paper introduces SVM Pooling (SVMP), a novel representation learning technique that treats video pooling as a Multiple Instance Learning (MIL) problem. By training a Support Vector Machine (SVM) to separate a video's frames from a negative bag of noise, the resulting hyperplane parameters serve as a highly discriminative, fixed-length video descriptor. SVMP achieves state-of-the-art results across diverse benchmarks, demonstrating particular strength in untrimmed video scenarios.

Introduction

Deep learning models for video action recognition typically process short clips or individual frames and then aggregate these predictions to classify the whole video. The standard ways to aggregate these features—average pooling or max pooling—are fundamentally flawed for complex video data. Average pooling gives equal weight to every frame, meaning the majority of a video where a person is stationary can dilute the critical moments where the action occurs. Max pooling, while focusing on peaks, is highly sensitive to outliers and artifacts.

Conceptual overview of SVM Pooling separating action-relevant frames from background actions.

The authors tackle the challenge of creating a fixed-length video representation that explicitly filters out irrelevant background noise and focuses only on the frames and feature dimensions that characterize the action. By learning a decision boundary for each video, the model can dynamically weight the importance of different temporal segments.

Methodology

The authors frame video pooling as a Multiple Instance Learning (MIL) problem. In this framework, the set of CNN features extracted from the frames of a target video constitutes a "Positive Bag," while a set of features known to be irrelevant forms a "Negative Bag."

The SVM Pooling pipeline: (i) frame extraction, (ii) feature mapping, (iii) per-video SVM training, and (iv) final video-level classification.

For each individual video, a binary SVM is trained to separate a certain fraction (e.g., 90%) of the positive bag from the negative bag. The parameters of this learned decision boundary—the weights and bias—become the final, pooled descriptor for the video. To handle the non-convexity of finding the optimal subset of frames, the authors employ a parameter-tuning heuristic that iteratively adjusts the SVM's regularization parameter ($C$). Furthermore, the SVMP operation is made differentiable using the implicit function theorem, allowing it to be integrated as a layer within end-to-end trainable CNNs.

Key Insights and Results

A surprising finding is that using CNN features generated from synthetic random white noise as the negative bag outperforms using actual video frames from unrelated classes. This forces the SVM to identify the most structured, action-correlated dimensions in the feature space. By using the hyperplane itself as a descriptor, the model leverages a max-margin framework that inherently assigns zero weight to non-support vectors, effectively ignoring irrelevant background frames.

SVMP demonstrates significant performance gains, outperforming standard pooling methods by 3% to 14% across eight diverse benchmarks. On the untrimmed Charades dataset, SVMP nearly doubles the classification mean Average Precision (mAP) compared to average pooling (26.3% vs 15.3%). When combined with the I3D network, it achieves state-of-the-art results on HMDB-51 (81.8%) and UCF-101 (98.5%).

Conclusion and Limitations

SVM Pooling represents a powerful paradigm shift in video representation, moving from simple aggregation to dynamic, data-driven descriptors. While it introduces some computational overhead due to per-video SVM optimization, its ability to isolate relevant data in untrimmed sequences makes it a robust solution for real-world video analysis. A current limitation is the sequence length dependency; SVMP requires a sufficient number of frames to learn a robust boundary, meaning its benefits are less pronounced on very short clips where traditional pooling remains competitive.

Summary updated

BibTeX
@article{wang2021discriminative,
  author = {Jue Wang and Anoop Cherian},
  title = {{Discriminative Video Representation Learning Using Support Vector Classifiers}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=86AJ_A_n8s},
}
Figure