Abstract
Previous approaches for video summarization mainly concentrate on finding the most diverse and representative visual contents as video summary without considering the user's preference. This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs and aims to generate a query-focused video summary. In this paper, we consider the task as a problem of computing similarity between video shots and query. To this end, we propose a method, named Convolutional Hierarchical Attention Network (CHAN), which consists of two parts: feature encoding network and query-relevance computing module. In the encoding network, we employ a convolutional network with local self-attention mechanism and query-aware global attention mechanism to learns visual information of each shot. The encoded features will be sent to query-relevance computing module to generate query-focused video summary. Extensive experiments on the benchmark dataset demonstrate the competitive performance and show the effectiveness of our approach.
Introduction
Video summarization is essential for navigating the vast amount of video content generated daily. While generic summarization identifies the most representative highlights, query-focused video summarization (QFVS) allows users to retrieve segments relevant to specific concepts, such as "Food" or "Sky."
Existing QFVS methods typically rely on Recurrent Neural Networks (RNNs) to model temporal dependencies. However, RNNs process data sequentially, leading to high computational costs and difficulty in capturing long-range dependencies in long-form videos (e.g., 3-5 hour egocentric videos). To address these limitations, we propose the Convolutional Hierarchical Attention Network (CHAN), a fully convolutional framework that enables parallel processing and efficient long-range context modeling.
Methodology: The CHAN Architecture
The CHAN framework shifts away from sequential modeling toward a parallelized, attention-driven approach.
The architecture consists of three primary components:
- Fully Convolutional Blocks: Utilizing 1D dilated convolutions, the model increases its temporal receptive field without a proportional increase in parameters. This allows the network to "see" broader segments of the video simultaneously.
- Hierarchical Attention Mechanism:
- Local Self-Attention: Captures dependencies between shots within a single video segment.
- Query-Aware Global Attention: Models the relationship between all video segments and the user's textual query (represented via GloVe embeddings).
- Query-Relevance Computing Module: Fuses local, global, and original visual features to compute a similarity score, ranking shots based on their relevance to the user's query.
Experimental Results
We evaluated CHAN on the UT Egocentric (UTE) dataset, which contains long-form daily life videos.
Quantitative Performance
CHAN achieved an average F1-score of 46.94%, outperforming previous state-of-the-art models like TPAN (46.05%). As shown in the performance breakdown, CHAN demonstrates consistent improvements across multiple video scenarios.
Efficiency Gains
The most significant advantage of CHAN is its inference speed. By replacing LSTMs with convolutional structures, the average inference time per video was reduced from 1.614s (TPAN) to 134.4ms. This represents a 91.6% reduction in processing time, making the model suitable for real-time applications.
Ablation Study
Ablation experiments revealed that the Query-Aware Global Attention module is critical. Removing it resulted in an 18.8% drop in performance, whereas removing Local Attention caused a smaller 7.84% decrease. This highlights the importance of global query context in the QFVS task.
Conclusion
The Convolutional Hierarchical Attention Network (CHAN) provides a high-performance, efficient solution for query-focused video summarization. By leveraging dilated convolutions and hierarchical attention, CHAN overcomes the limitations of sequential models, offering a 10x speedup while maintaining state-of-the-art accuracy. Future work will focus on scaling the model to more complex natural language queries and diverse video domains.
Summary updated