Convolutional Hierarchical Attention Network for Query-Focused Video Summarization

Shuwen Xiao, Zhou Zhao, Zijian Zhang, Xiaohui Yan, Min Yang

Abstract

Previous approaches for video summarization mainly concentrate on finding the most diverse and representative visual contents as video summary without considering the user's preference. This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs and aims to generate a query-focused video summary. In this paper, we consider the task as a problem of computing similarity between video shots and query. To this end, we propose a method, named Convolutional Hierarchical Attention Network (CHAN), which consists of two parts: feature encoding network and query-relevance computing module. In the encoding network, we employ a convolutional network with local self-attention mechanism and query-aware global attention mechanism to learns visual information of each shot. The encoded features will be sent to query-relevance computing module to generate query-focused video summary. Extensive experiments on the benchmark dataset demonstrate the competitive performance and show the effectiveness of our approach.

Introduction

Video summarization traditionally aims to identify the most significant "highlights" within a video. However, Query-Focused Video Summarization (QFVS) introduces a personalized dimension: the summary must align with specific user interests. For instance, a user searching for "food" within a lengthy travel vlog expects clips of meals rather than generic scenic highlights.

Example of a user query specifying concepts like "Food" and "Sky".

The primary technical challenges in QFVS involve modeling long-range dependencies in extensive video files and achieving effective multi-modal alignment between textual queries and visual segments. Traditional models often rely on Recurrent Neural Networks (RNNs) like LSTMs, which are computationally expensive for long videos and prone to the vanishing gradient problem.

Convolutional Hierarchical Attention Network (CHAN)

The proposed CHAN architecture shifts from sequential processing to a parallelized convolutional approach, treating summarization as a ranking problem. The model consists of three core components:

  1. 1D Fully Convolutional Blocks: Utilizing 1D dilated convolutions, the model encodes video shots with a large receptive field. This allows for the capture of long-range context without the sequential overhead of RNNs, enabling parallel processing of the entire video.
  2. Hierarchical Attention Mechanism:
    • Local Self-Attention: Captures immediate visual context within small video segments.
  3. Query-Aware Global Attention: Evaluates the relationship between all segments and the user's query, ensuring selected shots are specifically relevant to the requested concepts.
  4. Query-Relevance Computing Module: This module calculates similarity scores between encoded visual features and textual query embeddings (derived from GloVe vectors). Shots are ranked based on these scores to form the final summary.

Experimental Evaluation

The model was evaluated using the UT Egocentric (UTE) dataset, which contains four long-form first-person videos (3–5 hours each). The benchmark includes 46 unique queries, each comprising two concepts (e.g., "Book, Street"). The evaluation covers various scenarios, including cases where query concepts appear together, separately, or not at all, testing the model's robustness in both targeted and generic summarization.

Results and Insights

CHAN demonstrates state-of-the-art performance and significant efficiency gains:

  • Performance: The model achieved an average F1-score of 46.94%, surpassing the previous benchmark (TPAN) of 46.05%.
  • Efficiency: CHAN reduced inference time by 91.6%, dropping from 1.614 seconds to just 134.4 milliseconds per video.
  • Qualitative Success: The model effectively identifies diverse and representative shots matching complex queries.
A visual segment correctly identified as relevant to a "Food" query.

Key insights from ablation studies highlight the importance of the Global Attention module; its removal resulted in a nearly 9% drop in F1-score, confirming that global query awareness is vital for effective summarization.

Conclusion and Future Work

By replacing LSTMs with a fully convolutional structure, CHAN proves that sequential processing is not a prerequisite for understanding video context. Future research will focus on scaling the attention mechanism to handle complex natural language queries and improving zero-shot generalization for concepts not encountered during training. Expanding the evaluation to more diverse video genres remains a priority.

Summary updated

Abstract

Previous approaches for video summarization mainly concentrate on finding the most diverse and representative visual contents as video summary without considering the user's preference. This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs and aims to generate a query-focused video summary. In this paper, we consider the task as a problem of computing similarity between video shots and query. To this end, we propose a method, named Convolutional Hierarchical Attention Network (CHAN), which consists of two parts: feature encoding network and query-relevance computing module. In the encoding network, we employ a convolutional network with local self-attention mechanism and query-aware global attention mechanism to learns visual information of each shot. The encoded features will be sent to query-relevance computing module to generate query-focused video summary. Extensive experiments on the benchmark dataset demonstrate the competitive performance and show the effectiveness of our approach.
BibTeX
@inproceedings{xiao2020convolutional,
  author = {Shuwen Xiao and Zhou Zhao and Zijian Zhang and Xiaohui Yan and Min Yang},
  title = {{Convolutional Hierarchical Attention Network for Query-Focused Video Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=vbQXonVSg9q},
}
Figure