Query Twice: Dual Mixture Attention Meta Learning for Video Summarization

Junyan Wang, Yang Bai, Yang Long, Bingzhang Hu, Zhenhua Chai, Yu Guan, Xiaolin Wei

TL;DR

This paper introduces DMASum, a novel framework for video summarization that tackles the "softmax bottleneck" problem by using a "dual mixture attention" mechanism. By combining visual and sequential information with a meta-learning approach, DMASum significantly improves the ability to select representative frames, achieving human-level performance on benchmark datasets.

What Problem Does This Paper Address?

Automatic video summarization aims to create concise summaries of videos by selecting the most important frames. A common approach is to predict an "importance score" for each video segment. However, existing methods often rely on the softmax function to generate these scores. The authors identify a critical limitation of softmax: it struggles to retain high-rank representations for complex visual or sequential information, a phenomenon they term the "Softmax Bottleneck." This bottleneck can lead to the loss of crucial information, especially in long or complex videos, hindering the quality of the generated summaries. Furthermore, obtaining labeled data for video summarization is expensive, making it challenging to train models that generalize well to new videos, especially with limited training data.

Methods

The proposed DMASum model addresses these challenges through several key innovations:

  • Dual Mixture Attention (MoA) Layer: To overcome the softmax bottleneck, DMASum introduces a "Mixture of Attention" (MoA) layer. This layer employs a "Query Twice" strategy, incorporating both standard query-key attention and a novel self-query attention. This allows the model to capture not only initial relationships but also second-order changes within the video, effectively increasing its capacity to represent complex information. The MoA layer combines these attention mechanisms to produce a higher-rank attention distribution.
  • Dual-Channel Architecture: DMASum processes video content through two parallel channels:
  • Visual Channel: Utilizes deep Convolutional Neural Networks (CNNs) to extract visual features from individual frames.
  • Sequential Channel: Employs a Bidirectional Long Short-Term Memory (Bi-LSTM) network to capture temporal dependencies and sequential information from the visual features. These dual representations are then fused to create a more comprehensive understanding of the video.

  • Single-Video Meta Learning: To improve generalization, especially with limited training data, DMASum incorporates a meta-learning approach. Unlike traditional meta-learning that might train on multiple videos per task, DMASum introduces a "Single-Video Meta Learning" rule. This rule refrains the learner to focus on individual videos as distinct tasks, allowing the meta-learner to better adapt to the unique latent mechanisms of each video and thus achieve superior generalization.

Datasets

The paper evaluates DMASum on two widely used public datasets for video summarization:

  • SumMe: Contains 25 videos covering various events like sports and cooking, with durations ranging from 1 to 6.5 minutes.
  • TVSum: Consists of 50 videos downloaded from YouTube, spanning 10 categories, with lengths from 1 to 10 minutes.

Additionally, two auxiliary datasets, Open Video Project (OVP) and YouTube, were used to augment training data in specific experimental settings.

Key Insights and Ideas

  • The "Softmax Bottleneck" is Real and Detrimental: The paper provides both theoretical and empirical evidence that the standard softmax function in attention mechanisms is a significant limitation for video summarization, especially for complex or long videos.
  • Second-Order Attention is Crucial: By introducing self-query attention alongside query-key attention, the MoA layer can capture more nuanced relationships and second-order changes, breaking free from the limitations of traditional attention.
  • Individual Videos Have Unique Summarization "Mechanisms": The success of the "Single-Video Meta Learning" rule suggests that each video has its own distinct way of being summarized, and meta-learning strategies need to be tailored to this individuality for optimal generalization.
  • Human-like Attention is Achievable: The model's ability to achieve human-level performance on rank-order correlation metrics indicates that DMASum is effectively simulating human-like attention and summarization strategies.

Key Results

  • State-of-the-Art Performance: DMASum consistently outperforms existing state-of-the-art methods on both SumMe and TVSum datasets in terms of F1-score, achieving 54.3% on SumMe and 61.4% on TVSum in canonical settings.
  • Human-Level Correlation: The model achieves significantly higher Kendall's τ and Spearman's ρ correlation coefficients compared to other methods. Notably, on the TVSum dataset, DMASum even surpasses human annotator performance in these metrics (0.233 vs. 0.205 for τ, and 0.267 vs. 0.267 for ρ, though the paper states it's "even better" for τ, implying a slight edge).
  • Effective Bottleneck Resolution: Ablation studies show that replacing the MoA layer with standard softmax leads to a performance drop of 3.7% and 1.3% on SumMe and TVSum, respectively, confirming the effectiveness of MoA in resolving the softmax bottleneck.
  • Meta-Learning Necessity: Removing the meta-learning module results in a performance decrease of 2.7% on the smaller SumMe dataset, highlighting the importance of meta-learning for generalization.

Limitations and Future Work

  • Discrepancy in Label Formats: The paper notes that auxiliary datasets like OVP and YouTube have binary (0 or 1) importance scores, while SumMe and TVSum have scores within a range. This discrepancy can make meta-learning less effective, suggesting future work could focus on adapting the framework to handle such label format differences.
  • Cross-Dataset Generalization: While DMASum shows good performance across different settings, further improvements in cross-dataset generalization are a potential area for future research, possibly through an integrated framework.
  • Further Exploration of Meta-Learning Strategies: The paper suggests exploring more advanced meta-learning techniques or integrating them with other architectures.

Summary updated

Abstract

Video summarization aims to select representative frames to retain high-level information, which is usually solved by predicting the segment-wise importance score via a softmax function. However, softmax function suffers in retaining high-rank representations for complex visual or sequential information, which is known as the Softmax Bottleneck problem. In this paper, we propose a novel framework named Dual Mixture Attention (DMASum) model with Meta Learning for video summarization that tackles the softmax bottleneck problem, where the Mixture of Attention layer (MoA) effectively increases the model capacity by employing twice self-query attention that can capture the second-order changes in addition to the initial query-key attention, and a novel Single Frame Meta Learning rule is then introduced to achieve more generalization to small datasets with limited training sources. Furthermore, the DMASum significantly exploits both visual and sequential attention that connects local key-frame and global attention in an accumulative way. We adopt the new evaluation protocol on two public datasets, SumMe, and TVSum. Both qualitative and quantitative experiments manifest significant improvements over the state-of-the-art methods.
BibTeX
@inproceedings{wang2020query,
  author = {Junyan Wang and Yang Bai and Yang Long and Bingzhang Hu and Zhenhua Chai and Yu Guan and Xiaolin Wei},
  title = {{Query Twice: Dual Mixture Attention Meta Learning for Video Summarization}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=ny-MZhs1ylV},
}
Figure