Abstract
Selecting a representative thumbnail for short-form video is a critical yet challenging task due to the high similarity between frames. This paper presents a deep learning approach using a Siamese Convolutional Neural Network (CNN) that prioritizes facial quality and employs a ranking-based loss function. By leveraging a new 11,000-video dataset (SVCD), the proposed model outperforms standard image quality and video summarization baselines, achieving an accurate selection rate of over 60%.
Introduction
With the explosion of short-form video content on social media, selecting the perfect thumbnail is more important than ever—it’s the "first impression" that determines whether a user clicks. However, manually scrubbing through a video to find the best frame is tedious.
Existing solutions fall short for two main reasons. First, video summarization tools usually aim to create a shorter clip or a set of keyframes, rather than picking the single best image. Second, Image Quality Assessment (IQA) models often provide absolute scores. In a 5-second video, dozens of frames might have nearly identical high-quality scores, making it impossible for standard IQA to distinguish which one is truly the "best" representation of the moment. The authors tackle this challenge through fine-grained ranking within a sequence of highly similar frames.
Methodology
The authors propose a Siamese Convolutional Neural Network (CNN) architecture designed to learn relative preferences rather than absolute scores.
Ranking Objective
Instead of using standard regression, the model employs a Piecewise Ranking (PR) loss. This allows the network to learn that Frame A is better than Frame B, providing a more stable training signal for subjective human preferences.
Facial Feature Integration
Recognizing that humans are naturally drawn to faces, the model incorporates two specialized facial modules:
- Face Heatmap CNN: Encodes the size and location of faces within the frame.
- Face Quality CNN: A sub-network (based on SqueezeNet) that specifically evaluates the blurriness and aesthetic appeal of detected faces.
The network uses a GoogLeNet backbone. Features from the base image, the face heatmap, and the face quality module are concatenated and passed through fully connected layers to produce a final representativeness score.
Dataset: SVCD
The authors introduced the Stock Video Clip Dataset (SVCD), currently the largest of its kind for this task. It contains 11,000 diverse short videos (under 10 seconds) captured in daily life. Each video was sampled into 19 frames, which were then ranked by five distinct Amazon Mechanical Turk workers to establish a ground-truth "best" frame. Unlike existing datasets like SumMe or TVSum, SVCD provides dense, human-annotated rankings for short, single-event clips.
Key Results and Insights
The model was evaluated using Score Difference (SD), Rank Difference (RD), and Accurate Percentage (AP).
- Superiority over Baselines: The full model achieved an Accurate Percentage of 60.96%, significantly outperforming random selection (39.30%), middle frame selection (42.36%), and general image aesthetics models (~40%).
- Ranking > Regression: Predicting the difference between frames is more effective than predicting absolute quality scores because "best" is a relative term within a specific video context.
- Faces are Decisive: Specialized facial analysis is a major value-add. Even if a frame has high overall composition, a blurry or "mid-blink" face will cause a human to reject it.
- Sampling Strategy: "Best pairs sampling" (comparing the top-rated frame specifically against others) proved more effective than "all pairs sampling" during training.
Limitations and Future Work
The model currently struggles with videos containing dramatic scene changes or complex action where "representativeness" becomes highly subjective. While experiments with Recurrent Neural Networks (RNNs) to capture temporal flow did not yield significant improvements, future work may investigate better ways to incorporate the "story" or "action" of the video. Additionally, while the model aligns with average human preference, individual user tastes for thumbnails remain a factor for future personalization.
Summary updated