Abstract
This paper introduces BestShot, a novel task that shifts the focus of video retrieval from broad "moment" identification to high-precision "highlight frame" localization using complex language queries. While traditional Moment Retrieval (MR) identifies multi-second clips, many real-world applications—such as sports coaching, surgical analysis, and automated editing—require frame-level accuracy. To address this, the authors developed ShotVL, a model fine-tuned on two massive new datasets: ShotGPT4o and Image-SMPLText. ShotVL significantly outperforms existing state-of-the-art vision-language models, achieving a 53.6% top-1 accuracy on the BestShot benchmark, representing a 64% improvement over baseline architectures.
Introduction: The Granularity Gap in Video Understanding
Despite rapid advancements in video-language models, current systems often fail when precision is paramount. The authors identify two primary bottlenecks in existing methodologies:
- Granularity Gap: Most datasets rely on coarse descriptions (e.g., "a person jumping"), which are insufficient for distinguishing between the "peak of a jump" and the "landing."
- Pose Blindness: Models frequently lack the nuanced understanding of human body mechanics required to map complex textual descriptions (e.g., "left leg extended at 45 degrees") to specific visual frames.
The BestShot task addresses these issues by requiring models to retrieve a single, most representative frame based on detailed queries encompassing visual content, action stages, and specific human poses.
Methodology
ShotVL is built upon the InternVL-14B architecture, utilizing a Vision Transformer (ShotViT-6B) and a Large Language Model (LLaMA-7B) as its backbone. The training and inference pipelines are optimized for frame-level similarity:
- Furthest Point Sampling (FPS): To prevent the model from training on redundant, nearly identical adjacent frames, the researchers employed FPS based on pose error (MPJPE) to select the most visually and structurally distinct frames.
- Balanced Training Strategy: To maintain general semantic intelligence while learning specialized poses, the model was trained using a 1:5:5 ratio of pose data, GPT-generated descriptive data, and general image-captioning data (e.g., COCO).
- Inference Pipeline: During retrieval, the model calculates similarity scores for every frame against the query. The final "BestShot" is selected using either a simple
argmaxfunction or Non-Maximum Suppression (NMS) to ensure peak localization.
Data Innovation: ShotGPT4o and Image-SMPLText
A core contribution of this work is the creation of three large-scale resources:
- BestShot Benchmark: A high-quality evaluation set comprising 6,000 human-annotated queries across 60 action categories, categorized by visual content, fine-grained action, and human pose.
- ShotGPT4o Dataset: A training set of 600,000 frames featuring detailed captions and Q&A pairs generated by GPT-4o to provide rich semantic context.
- Image-SMPLText Dataset: A massive collection of 18.6 million pose descriptions. These were created by translating 3D human body model (SMPL) joint data into natural language, teaching the model to understand body mechanics as a primary feature.
Experimental Results
ShotVL establishes a new state-of-the-art (SOTA) for zero-shot frame retrieval. On the BestShot benchmark, it achieved 53.6% top-1 accuracy, a 64% improvement over the base InternVL model (32.6%).
Beyond the primary benchmark, ShotVL demonstrated remarkable cross-task performance:
- Temporal Action Localization: On the THUMOS14 dataset, ShotVL improved localization performance by 68% over InternVL.
- Generalization: The model maintained SOTA performance on general benchmarks like Flickr30k, proving that specialized pose training enhances rather than degrades general vision-language capabilities.
Discussion and Future Directions
While ShotVL excels at spatial and pose scrutiny, the authors note that it still faces challenges with actions requiring deep temporal context—such as distinguishing between "starting to sit" and "starting to stand" when the static frames appear identical. Additionally, automated pose descriptions (SMPLText) can occasionally describe occluded body parts, leading to model confusion. Future work will focus on integrating ShotVL with Video LLMs to enable conversational video editing and more robust temporal reasoning.
Summary updated