Exploring Global Diversity and Local Context for Video Summarization

Yingchao Pan, Ouhan Huang, Qinghao Ye, Zhongjin Li, Wenjiang Wang, Guodun Li, Yuxing Chen

Abstract

Video summarization aims to automatically generate a diverse and concise summary which is useful in large-scale video processing. Most of the methods tend to adopt self-attention mechanism across video frames, which fails to model the diversity of video frames. To alleviate this problem, we revisit the pairwise similarity measurement in self-attention mechanism and find that the existing inner-product affinity leads to discriminative features rather than diversified features. In light of this phenomenon, we propose global diverse attention which uses the squared Euclidean distance instead to compute the affinities. Moreover, we model the local contextual information by novel local contextual attention to remove the redundancy in the video. By combining these two attention mechanisms, a video SUMmarization model with Diversified Contextual Attention scheme is developed, namely SUM-DCA. Extensive experiments are conducted on benchmark data sets to verify the effectiveness and the superiority of SUM-DCA in terms of F-score and rank-based evaluation without any bells and whistles.

Introduction

Video summarization is the critical task of distilling long-form video content into a short, representative, and diverse set of keyframes or shots. While early approaches relied on Recurrent Neural Networks (RNNs), these models often struggled with long-range dependencies due to information decay over time. Recent attention-based models, such as Transformers, have addressed long-range modeling but introduced a new challenge: the "rich get richer" effect.

Standard self-attention mechanisms use dot-product similarity, which tends to favor frames with high-magnitude feature vectors. This results in discriminative features optimized for classification rather than the diversified features essential for a balanced summary. Furthermore, global attention often overlooks local temporal cues—the subtle relationships between adjacent frames that help identify the most essential moment within a specific scene. To address these issues, we introduce SUM-DCA, a framework that balances global diversity with local contextual relevance.

Methodology

The SUM-DCA framework utilizes a dual-attention architecture to process video features extracted via a Convolutional Neural Network (CNN). The core of the system is the Diversified Contextual Attention scheme, which combines Global Diverse Attention (GDA) and Local Contextual Attention (LCA) to produce a comprehensive feature representation $\tilde{X}$.

Overview of the SUM-DCA framework for video summarization.

Global Diverse Attention (GDA)

To overcome the limitations of dot-product attention, GDA replaces the standard inner-product affinity ($u^T v$) with the negative squared Euclidean distance ($-||u-v||^2$). By treating the feature space more democratically, this prevents high-magnitude frames from dominating the attention weights, thereby promoting a more diverse selection of summary candidates.

Detailed architecture of the Global Diversified Attention (GDA) module.

To maintain computational efficiency, the $L_2$ distance is decomposed using the expansion $(u-v)^2 = u^2 + v^2 - 2uv$. This allows the model to leverage fast GPU matrix multiplication while gaining the diversity benefits of Euclidean distance.

Local Contextual Attention (LCA)

While GDA handles global variety, LCA focuses on a small window of $2R$ adjacent frames. By utilizing relative positional embeddings, LCA captures the local "flow" of a scene. This helps the model identify the single most representative frame within a local cluster, effectively reducing redundancy by preventing the selection of multiple similar frames from the same shot.

Objective Functions

The model is trained using a multi-task loss function designed to optimize both accuracy and variety:

  1. Classification Loss ($\mathcal{L}_{cls}$): A supervised loss that aligns predicted importance scores with human-annotated ground truth.
  2. Repelling Loss ($\mathcal{L}_d$): A diversity-promoting loss that penalizes high cosine similarity between selected frames.
  3. Reconstruction Loss ($\mathcal{L}_r$): Ensures the generated summary retains enough information to reconstruct the original video's feature space.

Experimental Results

SUM-DCA was evaluated on standard benchmarks including SumMe (user-shot videos) and TVSum (YouTube videos across 10 categories). The model achieved state-of-the-art (SOTA) performance across multiple evaluation settings:

  • F-Score Performance: On the SumMe "Canonical" setting, SUM-DCA reached an F-score of 54.7%, significantly outperforming previous attention-based models like VASNet (49.7%).
  • Generalization: In "Transfer" settings—where the model is trained on one dataset and tested on another—SUM-DCA showed superior robustness, suggesting its attention mechanisms capture fundamental video structures rather than dataset-specific biases.
  • Human Alignment: Using rank correlation metrics (Kendall’s $\tau$ and Spearman’s $\rho$), the results demonstrated that SUM-DCA’s importance scores align more closely with human judgment than existing methods.

Conclusion

The SUM-DCA framework demonstrates that the choice of similarity function is a primary driver of diversity in video summarization. By replacing dot-product attention with Euclidean-based GDA and supplementing it with LCA, the model effectively balances global representativeness with local precision. Future work may explore learning these similarity functions dynamically and incorporating multi-modal features, such as audio, to further refine the identification of key video moments.

Summary updated

Abstract

Video summarization aims to automatically generate a diverse and concise summary which is useful in large-scale video processing. Most of the methods tend to adopt self-attention mechanism across video frames, which fails to model the diversity of video frames. To alleviate this problem, we revisit the pairwise similarity measurement in self-attention mechanism and find that the existing inner-product affinity leads to discriminative features rather than diversified features. In light of this phenomenon, we propose global diverse attention which uses the squared Euclidean distance instead to compute the affinities. Moreover, we model the local contextual information by novel local contextual attention to remove the redundancy in the video. By combining these two attention mechanisms, a video <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SUM</b> marization model with <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> iversified <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> ontextual <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ttention scheme is developed, namely SUM-DCA. Extensive experiments are conducted on benchmark data sets to verify the effectiveness and the superiority of SUM-DCA in terms of F-score and rank-based evaluation without any bells and whistles.
BibTeX
@inproceedings{pan2022exploring,
  author = {Yingchao Pan and Ouhan Huang and Qinghao Ye and Zhongjin Li and Wenjiang Wang and Guodun Li and Yuxing Chen},
  title = {{Exploring Global Diversity and Local Context for Video Summarization}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=V1-Nrx0IABu},
}
Figure