Abstract
The VQualA 2025 Challenge addressed the "cold-start" problem in social media recommendation systems by predicting short-video engagement using the novel SnapUGC dataset. Moving beyond traditional Video Quality Assessment (VQA), which focuses on technical fidelity, the competition explored methods to predict the Engagement Continuation Rate (ECR). The results demonstrate that Large Multimodal Models (LMMs) and ensembles incorporating visual, audio, and textual metadata significantly outperform traditional VQA methods, highlighting the importance of semantic content over simple pixel-level quality.
Introduction
Social media platforms face a persistent "cold-start" dilemma: when a new video is uploaded, the platform must determine its potential reach without historical performance data. The standard approach—distributing the video to a small test group to measure watch time—is often noisy, slow, and computationally expensive at scale.
Traditional Video Quality Assessment (VQA) has historically focused on technical distortions like blurriness or compression artifacts. However, in the context of user-generated content (UGC), technical quality is often secondary to engagement. A low-fidelity "meme" or a raw, authentic moment may capture viewer attention far more effectively than a high-definition but uninteresting landscape. The VQualA 2025 Challenge sought to bridge this gap by tasking participants with predicting the Engagement Continuation Rate (ECR)—the probability that a viewer watches past the first 5 seconds—using only the video's inherent content features.
The SnapUGC Dataset
The challenge introduced the SnapUGC Dataset, a significant advancement over previous benchmarks that typically relied on small groups of human raters in controlled environments.
- Scale and Diversity: The dataset comprises 120,651 short videos (ranging from 5 to 60 seconds) sourced from Snapchat Spotlight.
- Real-World Ground Truth: Unlike Mean Opinion Scores (MOS) derived from lab settings, SnapUGC metrics are calculated from over 2,000 real-world user interactions per video, providing a robust statistical foundation for engagement prediction.
- Multimodal Richness: Each entry includes the video file, audio track, and creator-provided metadata such as titles and descriptions.
Methodologies in Engagement Prediction
The competition saw a decisive shift away from hand-crafted features toward Large Multimodal Models (LMMs). The participating teams utilized several distinct strategies:
1. Large Multimodal Model (LMM) Ensembles
The top-performing teams, including ECNU-SJTU and IMCL-DAMO, leveraged ensembles of state-of-the-art models such as Qwen2.5-VL and Video-LLaMA2. These models are capable of processing video frames, audio tokens, and text descriptions in a unified architecture, allowing the system to "reason" about the narrative appeal and contextual relevance of the content.
2. Multi-Feature Fusion
Many participants combined disparate feature sets to capture the multifaceted nature of engagement:
- Semantic Features: Identifying the subject matter and narrative flow.
- Distortion/Technical Features: Assessing the baseline visual clarity.
- Aesthetic Features: Evaluating the artistic appeal and composition.
- Temporal/Spatial Rectifiers: Teams like HKUST-Cardiff implemented modules to adjust quality scores based on motion intensity and high-resolution spatial details.
3. The Efficacy of Metadata
One of the more surprising findings came from the "brucelyu17" team, which utilized a text-only approach. By applying an XGBoost regressor to video titles and descriptions alone, they achieved respectable results. This suggests that textual metadata remains a powerful signal for user intent and engagement, even in a video-centric medium.
Key Insights
The "5-Second Rule"
The challenge focused on the Engagement Continuation Rate (ECR), specifically the likelihood of a user watching past the 5-second mark. Analysis revealed a remarkably high correlation (0.928) between ECR and the Normalized Average Watch Percentage (NAWP). This confirms that the "hook" at the beginning of a video is the primary driver of overall retention.
VQA vs. Engagement
The results underscored that models trained on traditional VQA datasets (focused on fidelity) perform poorly when tasked with predicting engagement. Engagement is driven by content category, music choice, and "captivability" rather than just pixel-level perfection.
Performance Gains
The transition to LMM-based architectures provided a substantial performance boost. The baseline model (based on the Li et al. approach) achieved a Spearman Rank Order Correlation Coefficient (SROCC) of 0.657, while the winning ECNU-SJTU VQA team reached an SROCC of 0.707 and a final combined score of 0.710.
Conclusion and Future Directions
While LMMs have set a new standard for accuracy in engagement prediction, their computational cost remains a barrier to real-time, universal deployment. Future research must address the efficiency of these models to make them viable for high-traffic production environments. Furthermore, the current challenge focused on "global" popularity; however, engagement is inherently subjective. Integrating user-specific preferences and exploring long-term signals—such as shares, likes, and comments—represent the next frontier in short-video understanding.
Summary updated