Discriminatively Learned Hierarchical Rank Pooling Networks

Basura Fernando, Stephen Gould

Abstract

Video action recognition requires capturing the temporal evolution of visual information. Traditional pooling methods often ignore the chronological order of frames, while recurrent models can be computationally expensive. This paper introduces a framework that transforms "rank pooling"—a technique for summarizing video dynamics—into a deep, hierarchical, and end-to-end trainable architecture. By utilizing bilevel optimization to differentiate through the rank pooling solver, the authors enable the joint optimization of spatial features and temporal representations. The resulting Hierarchical Rank Pooling (HRP) network achieves state-of-the-art results on major activity recognition benchmarks, including HMDB51, Hollywood2, and UCF101.

Introduction

In computer vision, videos are distinguished from static images by the element of change. However, many early deep learning approaches for video recognition relied on "order-invariant" techniques like max or average pooling across frames. These methods produce the same feature vector regardless of frame sequence, failing to capture the essential chronology of an action. While Recurrent Neural Networks (RNNs) and LSTMs offer a solution for sequence modeling, they often introduce significant parameter overhead and require vast amounts of labeled data. This research addresses the need for an efficient, differentiable method to encode the "evolution of information" within video clips without the heavy overhead of traditional sequence models.

Methodology

The core of the proposed approach is Rank Pooling, which treats temporal encoding as a learning-to-rank problem. It identifies a vector that can chronologically order video frames based on their appearance.

End-to-End Discriminative Rank Pooling

A primary innovation of this work is making rank pooling differentiable. Previously an unsupervised step, the authors formulate a bilevel optimization problem. By applying Lemma 1 (Samuel & Tappen), they can calculate the gradient of the optimization problem's solution. This allows the network to backpropagate errors from the classification layer through the temporal pooling layer and back to the initial CNN filters, fine-tuning the entire network to extract features specifically optimized for capturing motion.

The end-to-end architecture allows backpropagation through the temporal pooling layer to fine-tune CNN features for motion capture.

Hierarchical Rank Pooling (HRP)

To capture complex, long-term activities, the authors introduce a hierarchical structure. Instead of pooling an entire video at once, HRP divides the sequence into overlapping segments. These segments are pooled to create "descriptors of dynamics," which are then pooled again at higher levels. This mirrors the spatial hierarchy of CNNs, where lower layers detect atomic motions and higher layers represent complex activities.

Hierarchical Rank Pooling (HRP) captures dynamics at multiple levels, evolving from atomic motions to higher-order dynamics.

Sign Expansion Root (SER)

The authors argue that standard ReLU non-linearities discard negative activations which might be useful for tracking how a feature disappears over time. The Sign Expansion Root (SER) transformation preserves both positive and negative components, effectively doubling the feature space to capture more nuanced temporal signals.

Experimental Results

The framework was evaluated on four standard benchmarks: HMDB51, Hollywood2, UCF101, and UCF-sports.

  • End-to-End Gains: On the UCF-sports dataset, end-to-end training yielded an accuracy of 87.1%, representing a ~21% improvement over non-end-to-end baselines.
  • State-of-the-Art Performance: HRP achieved 76.7% mAP on Hollywood2, 69.4% on HMDB51, and 93.6% on UCF101 (when integrated with optical flow and trajectory features).
  • Robustness: HRP consistently outperformed LSTMs and temporal pyramids in experiments, suggesting it provides a more robust summary of long-term dynamics into fixed-length vectors.

Conclusion and Future Work

This work demonstrates that the parameters of a model trained to order frames serve as a rich representation of video dynamics. By making this process differentiable and hierarchical, the authors provide a powerful tool for action recognition. The primary limitation is computational complexity, as calculating the gradient involves inverting Hessian matrices. Future work may involve kernelizing rank pooling to capture more complex non-linear dynamics or extending the framework to other sequence-based tasks like speech recognition.

Summary updated

Abstract

Rank pooling is a temporal encoding method that summarizes the dynamics of a video sequence to a single vector which has shown good results in human action recognition in prior work. In this work, we present novel temporal encoding methods for action and activity classification by extending the unsupervised rank pooling temporal encoding method in two ways. First, we present discriminative rank pooling in which the shared weights of our video representation and the parameters of the action classifiers are estimated jointly for a given training dataset of labelled vector sequences using a bilevel optimization formulation of the learning problem. When the frame level features vectors are obtained from a convolutional neural network (CNN), we rank pool the network activations and jointly estimate all parameters of the model, including CNN filters and fully-connected weights, in an end-to-end manner which we coined as end-to-end trainable rank pooled CNN. Importantly, this model can make use of any existing convolutional neural network architecture (e.g., AlexNet or VGG) without modification or introduction of additional parameters. Then, we extend rank pooling to a high capacity video representation, called hierarchical rank pooling. Hierarchical rank pooling consists of a network of rank pooling functions, which encode temporal semantics over arbitrary long video clips based on rich frame level features. By stacking non-linear feature functions and temporal sub-sequence encoders one on top of the other, we build a high capacity encoding network of the dynamic behaviour of the video. The resulting video representation is a fixed-length feature vector describing the entire video clip that can be used as input to standard machine learning classifiers. We demonstrate our approach on the task of action and activity recognition. We present a detailed analysis of our approach against competing methods and explore variants such as hierarchy depth and choice of non-linear feature function. Obtained results are comparable to state-of-the-art methods on three important activity recognition benchmarks with classification performance of 76.7% mAP on Hollywood2, 69.4% on HMDB51, and 93.6% on UCF101.
BibTeX
@article{fernando2017discriminatively,
  author = {Basura Fernando and Stephen Gould},
  title = {{Discriminatively Learned Hierarchical Rank Pooling Networks}},
  journal = {Int. J. Comput. Vis.},
  year = {2017},
  url = {https://openreview.net/forum?id=5m9sFGoPg4},
}
Figure