Best Frame Selection in a Short Video

Jian Ren, Xiaohui Shen, Zhe Lin, Radomír Mech

Abstract

Selecting a representative thumbnail for short-form video is a critical yet challenging task due to the high similarity between frames. This paper presents a deep learning approach using a Siamese Convolutional Neural Network (CNN) that prioritizes facial quality and employs a ranking-based loss function. By leveraging a new 11,000-video dataset (SVCD), the proposed model outperforms standard image quality and video summarization baselines, achieving an accurate selection rate of over 60%.

Introduction

With the explosion of short-form video content on social media, selecting the perfect thumbnail is more important than ever—it’s the "first impression" that determines whether a user clicks. However, manually scrubbing through a video to find the best frame is tedious.

Example frame sequences from the SVCD dataset illustrating the subtle differences between frames.

Existing solutions fall short for two main reasons. First, video summarization tools usually aim to create a shorter clip or a set of keyframes, rather than picking the single best image. Second, Image Quality Assessment (IQA) models often provide absolute scores. In a 5-second video, dozens of frames might have nearly identical high-quality scores, making it impossible for standard IQA to distinguish which one is truly the "best" representation of the moment. The authors tackle this challenge through fine-grained ranking within a sequence of highly similar frames.

Methodology

The authors propose a Siamese Convolutional Neural Network (CNN) architecture designed to learn relative preferences rather than absolute scores.

The proposed Siamese CNN architecture featuring face detection, heatmap generation, and quality estimation modules.

Ranking Objective

Instead of using standard regression, the model employs a Piecewise Ranking (PR) loss. This allows the network to learn that Frame A is better than Frame B, providing a more stable training signal for subjective human preferences.

Facial Feature Integration

Recognizing that humans are naturally drawn to faces, the model incorporates two specialized facial modules:

  • Face Heatmap CNN: Encodes the size and location of faces within the frame.
  • Face Quality CNN: A sub-network (based on SqueezeNet) that specifically evaluates the blurriness and aesthetic appeal of detected faces.

The network uses a GoogLeNet backbone. Features from the base image, the face heatmap, and the face quality module are concatenated and passed through fully connected layers to produce a final representativeness score.

Dataset: SVCD

The authors introduced the Stock Video Clip Dataset (SVCD), currently the largest of its kind for this task. It contains 11,000 diverse short videos (under 10 seconds) captured in daily life. Each video was sampled into 19 frames, which were then ranked by five distinct Amazon Mechanical Turk workers to establish a ground-truth "best" frame. Unlike existing datasets like SumMe or TVSum, SVCD provides dense, human-annotated rankings for short, single-event clips.

Key Results and Insights

The model was evaluated using Score Difference (SD), Rank Difference (RD), and Accurate Percentage (AP).

  • Superiority over Baselines: The full model achieved an Accurate Percentage of 60.96%, significantly outperforming random selection (39.30%), middle frame selection (42.36%), and general image aesthetics models (~40%).
  • Ranking > Regression: Predicting the difference between frames is more effective than predicting absolute quality scores because "best" is a relative term within a specific video context.
  • Faces are Decisive: Specialized facial analysis is a major value-add. Even if a frame has high overall composition, a blurry or "mid-blink" face will cause a human to reject it.
  • Sampling Strategy: "Best pairs sampling" (comparing the top-rated frame specifically against others) proved more effective than "all pairs sampling" during training.

Limitations and Future Work

The model currently struggles with videos containing dramatic scene changes or complex action where "representativeness" becomes highly subjective. While experiments with Recurrent Neural Networks (RNNs) to capture temporal flow did not yield significant improvements, future work may investigate better ways to incorporate the "story" or "action" of the video. Additionally, while the model aligns with average human preference, individual user tastes for thumbnails remain a factor for future personalization.

Summary updated

Abstract

People usually take short videos to record meaningful moments in their lives. However, selecting the most representative frame, which not only has high image visual quality but also captures video content, from a short video to share or keep is a time-consuming process for one may need to manually go through all the frames in a video to make a decision. In this paper, we introduce the problem of the best frame selection in a short video and aim to solve it automatically. Towards this end, we collect and will release a diverse large-scale short video dataset that includes 11, 000 videos shoot in our daily life. All videos are assumed to be short (e.g., a few seconds) and each video has human-annotated of the best frame. Then we introduce a deep convolutional neural network (CNN) based approach with ranking objective to automatically pick the best frame from frame sequences extracted via short videos. Additionally, we propose new evaluation metrics, especially for the best frame selection. In experiments, we show our approach outperforms various other methods significantly.
BibTeX
@inproceedings{ren2020best,
  author = {Jian Ren and Xiaohui Shen and Zhe Lin and Radomír Mech},
  title = {{Best Frame Selection in a Short Video}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093615},
}
Figure