Discriminative Video Representation Learning Using Support Vector Classifiers.

Jue Wang, Anoop Cherian

TL;DR

Instead of using simple heuristics like average or max pooling to summarize video clips, this paper introduces SVM Pooling (SVMP). By training a support vector machine to separate "useful" action features from a "negative bag" of noise and irrelevant data, the authors use the resulting hyperplane parameters as a highly discriminative descriptor for the entire video.

What Problem Does This Paper Address?

Deep learning models for action recognition typically process videos as a series of short clips. To classify a full video, these clip-level predictions must be "pooled" into a single representation. The industry standards—average and max pooling—are simple but flawed. Average pooling treats every frame as equally important, even though many frames are just background or common across different actions (e.g., a person standing still before kicking a ball). Max pooling, while better at picking out salient features, is notoriously sensitive to outliers.

The authors argue that a video representation should be discriminative: it should explicitly filter out the "noise" of background frames and focus on the specific features that characterize the action.

Methods

The core innovation is casting video pooling as a Multiple Instance Learning (MIL) problem solved via a Support Vector Machine (SVM).

  1. The Bags: For a given video, the authors create a "positive bag" containing all its clip-level CNN features. They also define a "negative bag" consisting of irrelevant features (e.g., features from unrelated datasets or even CNN outputs generated from random noise).
  2. The Objective: They learn a (possibly non-linear) hyperplane that separates at least a fraction ($\eta$) of the positive features from the negative bag.
  3. The Descriptor: Crucially, they don't just use the SVM to classify; they use the parameters of the learned hyperplane ($w$ and $b$) as the new, fixed-length descriptor for the entire video.
  4. End-to-End Learning: By applying the implicit function theorem, the authors derive gradients for this argmin-based pooling layer, allowing it to be integrated directly into deep neural networks for end-to-end training.
Fig. 2. Illustration of our SVM Pooling scheme. (i) Extraction of frames from videos, (ii) Converting frames f into feature x, (iii) Learning decision boundary w from feature x, and (iv) Using w as video descriptor.

The authors also explore extensions like Non-linear SVMP (using kernel maps) and Temporally-Ordered SVMP, which adds constraints to ensure the descriptor captures the chronological progression of the action.

Datasets

The researchers evaluated SVMP across eight diverse benchmarks to prove its versatility:

  • Action Recognition: HMDB-51, UCF-101, and the massive Kinetics-600.
  • Untrimmed/Multi-action: Charades.
  • Skeleton-based Action: MSR Action3D and NTU-RGBD.
  • Image Set Verification: PubFig (face recognition).
  • Dynamic Texture Recognition: YUP++.

Key Insights and Ideas

  • Noise as a Catalyst: One of the most counter-intuitive findings is that using random white noise as the negative bag often yields better results than using real "background" videos. The authors suggest that noise activates combinations of CNN filters that are rarely co-activated by real images, forcing the SVM to find highly specific, non-linear boundaries for the actual action features.
  • Hyperplane as Weighted Average: Because the hyperplane parameters are determined by support vectors, SVMP can be viewed as a form of "intelligent" weighted average pooling. It assigns zero weight to non-support vectors (irrelevant frames) and focuses entirely on the most discriminative data points.
  • Robustness to Untrimmed Video: SVMP shines in untrimmed datasets like Charades. Because it is designed to find a subset of useful features, it naturally ignores the long stretches of "nothing happening" that typically degrade the performance of average pooling.
Intuition for adding a negative bag: (a) without noise, many hyperplanes are possible; (b) with noise (green dots), the model is forced to find a boundary that isolates truly useful features.

Key Results

  • Performance Gains: SVMP consistently outperformed standard pooling methods by 3–14% across various datasets.
  • State-of-the-Art (SOTA): The method achieved SOTA results on nearly all tested benchmarks, typically improving accuracy by 1–5% over previous top-tier models.
  • Efficiency: While more complex than average pooling, SVMP is significantly faster to generate than other advanced methods like Rank Pooling.
Comparison of generation time: SVMP (Decision Boundary) is much faster than Rank Pooling or Dynamic Images as the number of frames increases.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance depends on choosing the right fraction $\eta$ and regularization $C$. While the authors provide a heuristic for tuning these, it adds a layer of complexity to the training pipeline.
  • Negative Bag Selection: While random noise works surprisingly well, the "optimal" negative bag might vary depending on the specific task or CNN architecture used.
  • Future Directions: The authors suggest that exploring more complex latent variable models within the MIL framework could further improve the localization of actions in very long, complex video streams.

Summary updated

Abstract

Most popular deep models for action recognition in videos generate independent predictions for short clips, which are then pooled heuristically to assign an action label to the full video segment. As not all frames may characterize the underlying action---many are common across multiple actions---pooling schemes that impose equal importance on all frames might be unfavorable. In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action. To identify these useful features, we resort to a negative bag consisting of features that are known to be irrelevant, for example, they are sampled either from datasets that are unrelated to our actions of interest or are CNN features produced via random noise as input. With the features from the video as a positive bag and the irrelevant features as the negative bag, we cast an objective to learn a (nonlinear) hyperplane that separates the unknown useful features from the rest in a multiple instance learning formulation within a support vector machine setup. We use the parameters of this separating hyperplane as a descriptor for the full video segment. Since these parameters are directly related to the support vectors in a max-margin framework, they can be treated as a weighted average pooling of the features from the bags, with zero weights given to non-support vectors. Our pooling scheme is end-to-end trainable within a deep learning framework. We report results from experiments on eight computer vision benchmark datasets spanning a variety of video-related tasks and demonstrate state-of-the-art performance across these tasks.
BibTeX
@article{wang2021discriminative,
  author = {Jue Wang and Anoop Cherian},
  title = {{Discriminative Video Representation Learning Using Support Vector Classifiers.}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=86AJ_A_n8s},
}

Loading related content...

Figure