Multiple Pairwise Ranking Networks for Personalized Video Summarization

Yassir Saquil, Da Chen, Yuan He, Chuan Li, Yong-Liang YangICCV 2021 · OpenReview

TL;DR

Video summarization is inherently subjective—what one person finds "important" in a video, another might find boring. This paper introduces a Multi-ranker model that moves away from "one-size-fits-all" summaries by training multiple pairwise ranking networks to respect specific user preferences (like specific sports actions), allowing for both personalized and high-quality global video highlights.

What Problem Does This Paper Address?

Most video summarization research treats the task as an objective problem: find the "best" frames to represent a video. However, "best" is in the eye of the beholder. A basketball coach might want a summary of defensive rotations, while a fan wants a highlight reel of slam dunks.

Existing supervised models typically produce a single global summary, which is restrictive. While some "personalized" methods exist, they often require a user's entire viewing history or complex natural language queries. The authors identify a gap: we need a flexible system that can condition summaries on predefined semantic categories (preferences) while still maintaining the ability to generate a standard global summary.

Methods

The authors frame summarization as a pairwise ranking problem. Instead of predicting an absolute "importance score" for a segment, the model learns to determine which of two segments is more important.

The Multi-ranker Architecture

The core innovation is the Multi-ranker, a collection of sub-ranking networks $\{R_j\}$ that are trained jointly.

  1. Local Ranking: Each sub-ranker is responsible for a specific "preference" (e.g., "Vaulting" in a gymnastics video). It learns to rank segments containing that specific action higher than others.
  2. Global Ranking: To ensure the model can still produce a general summary, the authors use a "max-aggregation" strategy. The global importance score for a segment is defined as the maximum score across all sub-rankers.
  3. Joint Loss Function: The training loss is a weighted balance ($\lambda$) between the global ranking accuracy and the local preference-specific ranking. This ensures that the sub-rankers don't just work in isolation but contribute to a coherent global understanding of the video.

At inference time, a user can select a subset of preferences, and the model generates a summary by aggregating the scores of the corresponding sub-rankers.

Datasets

The researchers evaluated their approach on three datasets:

  • TVSum & SumMe: The "gold standards" for video summarization. Since these lack explicit category labels for segments, the authors used k-means clustering on visual features to create "pseudo-preferences."
  • FineGym: A fine-grained action recognition dataset of gymnastics. The authors repurposed this as a summarization benchmark by using the action labels (e.g., Balance Beam, Floor Exercise) as the ground-truth preferences. This is a significant step toward more semantic, action-based summarization.

Key Insights and Ideas

  • The "Max" is Key: Using the maximum score across sub-rankers as the global score is a clever way to ensure that if any preference finds a segment important, the global summarizer considers it too.
  • Ranking > Regression: The paper reinforces that ranking-based losses are often more effective for subjective tasks than regressing to a mean importance score, as they better capture the relative importance that humans use when editing video.
  • Semantic Preferences: By using FineGym, the authors demonstrate that "preferences" don't have to be abstract; they can be concrete, human-understandable action categories.

Key Results

  • Global Performance: On the TVSum dataset, the Multi-ranker achieved a Kendall’s $\tau$ of 0.1750, which is competitive with state-of-the-art supervised methods like VASNet, proving that adding personalization doesn't degrade global quality.
  • Personalization Utility: In the FineGym experiments, the Multi-ranker significantly outperformed a standard "global" ranker when evaluated against specific user preferences. For example, when looking for a specific action, the Multi-ranker's correlation was often 10-20% higher than the baseline (see Table 2 in the paper).
  • User Study: A study with 40 subjects showed that ~85% of users felt the personalized summaries provided better control and satisfaction compared to a single global summary.

Limitations and Future Work

  • Predefined Categories: The model currently relies on predefined categories. If a user has a preference that wasn't in the training set (or wasn't discovered via clustering), the model can't explicitly cater to it.
  • Temporal Coherence: Like many frame- or segment-level ranking models, this approach doesn't explicitly optimize for the "flow" or "storytelling" aspect of a summary. It picks the most important parts but doesn't necessarily ensure they transition well.
  • Scalability of Preferences: While a 1-layer network for each ranker is computationally cheap, scaling this to hundreds of very niche preferences might require a more hierarchical or embedding-based approach.

The authors suggest that future work should explore more complex criteria like diversity and visual coherence within these personalized frameworks.

Summary updated

Abstract

In this paper, we investigate video summarization in the supervised setting. Since video summarization is subjective to the preference of the end-user, the design of a unique model is limited. In this work, we propose a model that provides personalized video summaries by conditioning the summarization process with predefined categorical user labels referred to as preferences. The underlying method is based on multiple pairwise rankers (called Multi-ranker), where the rankers are trained jointly to provide local summaries as well as a global summarization of a given video. In order to demonstrate the relevance and applications of our method in contrast with a classical global summarizer, we conduct experiments on multiple benchmark datasets, notably through a user study and comparisons with the state-of-art methods in the global video summarization task.
BibTeX
@inproceedings{saquil2021multiple,
  author = {Yassir Saquil and Da Chen and Yuan He and Chuan Li and Yong-Liang Yang},
  title = {{Multiple Pairwise Ranking Networks for Personalized Video Summarization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=2t9qNLWoet},
}
Figure