ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries

Wangyu Xue, Chen Qian, Jiayi Wu, Yang Zhou, Wentao Liu, Ju Ren, Siming Fan, Yaoxue Zhang

Abstract

This paper introduces BestShot, a novel task that shifts the focus of video retrieval from broad "moment" identification to high-precision "highlight frame" localization using complex language queries. While traditional Moment Retrieval (MR) identifies multi-second clips, many real-world applications—such as sports coaching, surgical analysis, and automated editing—require frame-level accuracy. To address this, the authors developed ShotVL, a model fine-tuned on two massive new datasets: ShotGPT4o and Image-SMPLText. ShotVL significantly outperforms existing state-of-the-art vision-language models, achieving a 53.6% top-1 accuracy on the BestShot benchmark, representing a 64% improvement over baseline architectures.

Introduction: The Granularity Gap in Video Understanding

Despite rapid advancements in video-language models, current systems often fail when precision is paramount. The authors identify two primary bottlenecks in existing methodologies:

  1. Granularity Gap: Most datasets rely on coarse descriptions (e.g., "a person jumping"), which are insufficient for distinguishing between the "peak of a jump" and the "landing."
  2. Pose Blindness: Models frequently lack the nuanced understanding of human body mechanics required to map complex textual descriptions (e.g., "left leg extended at 45 degrees") to specific visual frames.

The BestShot task addresses these issues by requiring models to retrieve a single, most representative frame based on detailed queries encompassing visual content, action stages, and specific human poses.

Example of the BestShot task, illustrating the high granularity of visual, action, and pose descriptions required for precise frame retrieval.

Methodology

ShotVL is built upon the InternVL-14B architecture, utilizing a Vision Transformer (ShotViT-6B) and a Large Language Model (LLaMA-7B) as its backbone. The training and inference pipelines are optimized for frame-level similarity:

  • Furthest Point Sampling (FPS): To prevent the model from training on redundant, nearly identical adjacent frames, the researchers employed FPS based on pose error (MPJPE) to select the most visually and structurally distinct frames.
  • Balanced Training Strategy: To maintain general semantic intelligence while learning specialized poses, the model was trained using a 1:5:5 ratio of pose data, GPT-generated descriptive data, and general image-captioning data (e.g., COCO).
  • Inference Pipeline: During retrieval, the model calculates similarity scores for every frame against the query. The final "BestShot" is selected using either a simple argmax function or Non-Maximum Suppression (NMS) to ensure peak localization.

Data Innovation: ShotGPT4o and Image-SMPLText

A core contribution of this work is the creation of three large-scale resources:

  1. BestShot Benchmark: A high-quality evaluation set comprising 6,000 human-annotated queries across 60 action categories, categorized by visual content, fine-grained action, and human pose.
  2. ShotGPT4o Dataset: A training set of 600,000 frames featuring detailed captions and Q&A pairs generated by GPT-4o to provide rich semantic context.
  3. Image-SMPLText Dataset: A massive collection of 18.6 million pose descriptions. These were created by translating 3D human body model (SMPL) joint data into natural language, teaching the model to understand body mechanics as a primary feature.

Experimental Results

ShotVL establishes a new state-of-the-art (SOTA) for zero-shot frame retrieval. On the BestShot benchmark, it achieved 53.6% top-1 accuracy, a 64% improvement over the base InternVL model (32.6%).

Performance comparison of ShotVL against InternVL and CLIP across multiple benchmarks, highlighting significant gains in zero-shot frame retrieval.

Beyond the primary benchmark, ShotVL demonstrated remarkable cross-task performance:

  • Temporal Action Localization: On the THUMOS14 dataset, ShotVL improved localization performance by 68% over InternVL.
  • Generalization: The model maintained SOTA performance on general benchmarks like Flickr30k, proving that specialized pose training enhances rather than degrades general vision-language capabilities.

Discussion and Future Directions

While ShotVL excels at spatial and pose scrutiny, the authors note that it still faces challenges with actions requiring deep temporal context—such as distinguishing between "starting to sit" and "starting to stand" when the static frames appear identical. Additionally, automated pose descriptions (SMPLText) can occasionally describe occluded body parts, leading to model confusion. Future work will focus on integrating ShotVL with Video LLMs to enable conversational video editing and more robust temporal reasoning.

Summary updated

Abstract

Existing research on human-centric video understanding typically focuses on analyzing specific moments or entire videos. However, many applications require higher precision at the frame level. In this work, we propose a novel task, BestShot, which aims to locate highlight frames within human-centric videos through language queries. This task requires not only a deep semantic understanding of human actions but also precise temporal localization. To support this task, we introduce the BestShot Benchmark. The benchmark is meticulously constructed by combining human-annotated highlight frames, duration labels and detailed textual descriptions. These descriptions cover three critical elements: (1) Visual content; (2) Fine-grained actions; and (3) Human pose descriptions. Together, these elements provide the necessary precision to identify the exact highlight frames in videos. To tackle this problem, we have collected two distinct datasets: (i) ShotGPT4o Dataset, which is algorithmically generated by GPT-4o and (ii) Image-SMPLText Dataset, which features large-scale and accurate per-frame pose descriptions using PoseScript and existing pose estimation datasets. Based on these datasets, we present a strong baseline model, ShotVL, fine-tuned from InternVL, specifically for BestShot. We highlight the impressive zero-shot capabilities of our model and offer comparative analyses with existing state-of-the-art (SOTA) models. ShotVL demonstrates a significant 64% improvement over InternVL on the BestShot Benchmark and a notable 68% improvement on the THUMOS14 Benchmark, while maintaining SOTA performance in general image classification and retrieval.
BibTeX
@inproceedings{xue2025shotvl,
  author = {Wangyu Xue and Chen Qian and Jiayi Wu and Yang Zhou and Wentao Liu and Ju Ren and Siming Fan and Yaoxue Zhang},
  title = {{ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.32979},
}
Figure