Traditional video summarization operates on a flawed assumption: that "importance" is an intrinsic property of a video frame. Early automated methods relied heavily on low-level visual features like color diversity, motion dynamics, or generic visual saliency to extract keyframes or highlights (Video Key Frame Extraction through Canonical Correlation Analysis and Graph Modularity).
However, as Aidean Sharghi et al. point out in Query-Focused Extractive Video Summarization, this "one-size-fits-all" approach ignores the deeply subjective nature of human interest. What makes a segment important depends entirely on the viewer's intent, expertise, or context. For instance, Esa Rahtu notes that a novice watching a sports video might favor high-energy visual exchanges, whereas an expert looks for technical precision (Summarization of User-Generated Sports Video by Using Deep Action Recognition Features). Because visual complexity does not equal personal relevance, a quiet, specific interaction might be far more critical to a user than a visually explosive but irrelevant scene (Gaze-enabled egocentric video summarization via constrained submodular maximization).
To bridge this gap, modern architectures have shifted from query-agnostic saliency to subjective importance modeling, using explicit queries, historical preferences, and real-time interaction to customize content.
Conditioning Saliency on Natural Language Queries
The most direct way to capture user intent is through explicit natural language queries. Instead of computing a static importance score for each frame, query-focused summarization models compute a conditional relevance score. This requires mapping textual concepts to dynamic visual segments.
To achieve this, Jinsoo Choi et al. proposed using a joint vision-language embedding space to bridge the gap between static text descriptions and dynamic video content (Textually Customized Video Summaries). Once both modalities share an embedding space, the summarization task can be framed as a submodular optimization problem. This mathematical framework balances two competing forces: maximizing the semantic relevance of selected segments to the user's query, while minimizing redundancy to ensure the summary remains visually diverse and representative of the broader video narrative (Query-adaptive Video Summarization via Quality-aware Relevance Estimation, Enhancing Video Summarization via Vision-Language Embedding).
[ Raw Video Frames ] ───► [ Visual Encoder ] ───┐
▼
[ Joint Embedding Space ] ───► [ Submodular Optimization ] ───► [ Personalized Summary ]
▲
[ User Text Query ] ───► [ Text Encoder ] ─────┘
To handle long-range dependencies and speed up processing, Shuwen Xiao et al. introduced the Convolutional Hierarchical Attention Network (CHAN) in Query-Biased Self-Attentive Network for Query-Focused Video Summarization. CHAN uses local self-attention to capture short-term video context and a query-aware global attention mechanism to pull out segments that specifically match the user's textual request. By explicitly biasing the attention weights with the query embedding, the model effectively filters out generally interesting highlights in favor of query-specific moments.
Learning Latent Preferences from User History
Users do not always provide explicit queries. When intent is implicit, models must infer subjective importance by analyzing historical behavior. The core intuition behind this approach is that subjective interest behaves like a measurable "style" that can be learned and applied to standard video editing pipelines.
In Adaptive Video Highlight Detection by Learning from User History, Mrigank Rochan et al. demonstrated how to personalize highlight detection using a user's past interactions, such as GIFs they previously created. They introduced a Temporal-Adaptive Instance Normalization (T-AIN) layer. Rather than retraining the entire network, T-AIN dynamically shifts the intermediate feature distributions of a generic highlight detection network toward a user-specific style, effectively "nudging" the model's predictions to align with individual tastes.
Building on this idea, Hang Wang et al. developed PAC-Net, which treats a user's viewing history as a collection of diverse "mini-interests" (PAC-Net: Highlight Your Video via History Preference Modeling). Instead of training a massive, rigid model for every user, PAC-Net dynamically generates custom lightweight classifiers tailored to the user's historical preferences. This allows the system to adapt its highlight predictions on the fly while maintaining a minimal computational footprint.
Resolving Subjectivity via Interactive and Implicit Feedback
When neither a text query nor a rich user history is available, systems can resolve the ambiguity of subjective importance through real-time interaction or implicit physiological signals.
The authors of Active Video Summarization: Customized Summaries via On-line Interaction with the User introduced an active learning approach called Active Video Summarization (AVS). Instead of guessing what a user wants, the system sequentially selects and presents a small number of highly informative, binary questions (e.g., "Are you interested in this object or scene?"). By updating a user preference profile on-the-fly based on these responses, the model converges on a highly personalized summary within just a few interactions.
Alternatively, implicit feedback like gaze tracking can bypass the need for active user input entirely. In Gaze-enabled egocentric video summarization via constrained submodular maximization, Jia Xu et al. leveraged egocentric gaze data to guide video summarization. Gaze fixations serve as a direct proxy for user intent, allowing a submodular maximization framework to prioritize frames that the user actually focused on. This effectively separates personal cognitive importance from raw visual saliency, ensuring that the final summary reflects the wearer's actual goals rather than just the most visually complex parts of the environment.