VQualA 2025 Challenge on Engagement Prediction for Short Videos: Methods and Results

Dasong Li, Sizhuo Ma, Hang Hua, Wenjie Li, Jian Wang, Cunliu Zhou, Fengbin Guan, Xin Li, Zihao Yu, Yiting Lu, Ru-Ling Liao, Yan Ye, Zhibo Chen, Wei Sun, Linhan Cao, Yuqin Cao, Weixia Zhang, Wen Wen, Kaiwei Zhang, Zijian Chen, Fangfang Lu, Xiongkuo Min, Guangtao Zhai, Erjia Xiao, Lingfeng Zhang, Zhenjie Su, Hao Cheng, Yu Liu, Renjing Xu, Long Chen, Xiaoshuai Hao, Jianqin Wu, Jianain Wu, Xuxu Wang Oian Yu, Bo Hu, Weiwei Wang, Pinxin Liu, Yunlong Tang, Luchuan Song, Jinxi He, Jiaru Wu, Hanjia Lyu

Abstract

The VQualA 2025 Challenge addressed the "cold-start" problem in social media recommendation systems by predicting short-video engagement using the novel SnapUGC dataset. Moving beyond traditional Video Quality Assessment (VQA), which focuses on technical fidelity, the competition explored methods to predict the Engagement Continuation Rate (ECR). The results demonstrate that Large Multimodal Models (LMMs) and ensembles incorporating visual, audio, and textual metadata significantly outperform traditional VQA methods, highlighting the importance of semantic content over simple pixel-level quality.

Introduction

Social media platforms face a persistent "cold-start" dilemma: when a new video is uploaded, the platform must determine its potential reach without historical performance data. The standard approach—distributing the video to a small test group to measure watch time—is often noisy, slow, and computationally expensive at scale.

Traditional Video Quality Assessment (VQA) has historically focused on technical distortions like blurriness or compression artifacts. However, in the context of user-generated content (UGC), technical quality is often secondary to engagement. A low-fidelity "meme" or a raw, authentic moment may capture viewer attention far more effectively than a high-definition but uninteresting landscape. The VQualA 2025 Challenge sought to bridge this gap by tasking participants with predicting the Engagement Continuation Rate (ECR)—the probability that a viewer watches past the first 5 seconds—using only the video's inherent content features.

The SnapUGC Dataset

The challenge introduced the SnapUGC Dataset, a significant advancement over previous benchmarks that typically relied on small groups of human raters in controlled environments.

  • Scale and Diversity: The dataset comprises 120,651 short videos (ranging from 5 to 60 seconds) sourced from Snapchat Spotlight.
  • Real-World Ground Truth: Unlike Mean Opinion Scores (MOS) derived from lab settings, SnapUGC metrics are calculated from over 2,000 real-world user interactions per video, providing a robust statistical foundation for engagement prediction.
  • Multimodal Richness: Each entry includes the video file, audio track, and creator-provided metadata such as titles and descriptions.

Methodologies in Engagement Prediction

The competition saw a decisive shift away from hand-crafted features toward Large Multimodal Models (LMMs). The participating teams utilized several distinct strategies:

1. Large Multimodal Model (LMM) Ensembles

The top-performing teams, including ECNU-SJTU and IMCL-DAMO, leveraged ensembles of state-of-the-art models such as Qwen2.5-VL and Video-LLaMA2. These models are capable of processing video frames, audio tokens, and text descriptions in a unified architecture, allowing the system to "reason" about the narrative appeal and contextual relevance of the content.

2. Multi-Feature Fusion

Many participants combined disparate feature sets to capture the multifaceted nature of engagement:

  • Semantic Features: Identifying the subject matter and narrative flow.
  • Distortion/Technical Features: Assessing the baseline visual clarity.
  • Aesthetic Features: Evaluating the artistic appeal and composition.
  • Temporal/Spatial Rectifiers: Teams like HKUST-Cardiff implemented modules to adjust quality scores based on motion intensity and high-resolution spatial details.

3. The Efficacy of Metadata

One of the more surprising findings came from the "brucelyu17" team, which utilized a text-only approach. By applying an XGBoost regressor to video titles and descriptions alone, they achieved respectable results. This suggests that textual metadata remains a powerful signal for user intent and engagement, even in a video-centric medium.

Key Insights

The "5-Second Rule"

The challenge focused on the Engagement Continuation Rate (ECR), specifically the likelihood of a user watching past the 5-second mark. Analysis revealed a remarkably high correlation (0.928) between ECR and the Normalized Average Watch Percentage (NAWP). This confirms that the "hook" at the beginning of a video is the primary driver of overall retention.

VQA vs. Engagement

The results underscored that models trained on traditional VQA datasets (focused on fidelity) perform poorly when tasked with predicting engagement. Engagement is driven by content category, music choice, and "captivability" rather than just pixel-level perfection.

Performance Gains

The transition to LMM-based architectures provided a substantial performance boost. The baseline model (based on the Li et al. approach) achieved a Spearman Rank Order Correlation Coefficient (SROCC) of 0.657, while the winning ECNU-SJTU VQA team reached an SROCC of 0.707 and a final combined score of 0.710.

Conclusion and Future Directions

While LMMs have set a new standard for accuracy in engagement prediction, their computational cost remains a barrier to real-time, universal deployment. Future research must address the efficiency of these models to make them viable for high-traffic production environments. Furthermore, the current challenge focused on "global" popularity; however, engagement is inherently subjective. Integrating user-specific preferences and exploring long-term signals—such as shares, likes, and comments—represent the next frontier in short-video understanding.

Summary updated

Abstract

This paper presents an overview of the VQualA 2025 Challenge on Engagement Prediction for Short Videos, held in conjunction with ICCV 2025. The challenge focuses on understanding and modeling the popularity of user-generated content (UGC) short videos on social media platforms. To support this goal, the challenge uses a new short-form UGC dataset featuring engagement metrics derived from real-world user interactions. This objective of the Challenge is to promote robust modeling strategies that capture the complex factors influencing user engagement. Participants explored a variety of multi-modal features, including visual content, audio, and metadata provided by creators. The challenge attracted 97 participants and received 15 valid test submissions, contributing significantly to progress in short-form UGC video engagement prediction.
BibTeX
@misc{li2025vquala,
  author = {Dasong Li and Sizhuo Ma and Hang Hua and Wenjie Li and Jian Wang and Cunliu Zhou and Fengbin Guan and Xin Li and Zihao Yu and Yiting Lu and Ru-Ling Liao and Yan Ye and Zhibo Chen and Wei Sun and Linhan Cao and Yuqin Cao and Weixia Zhang and Wen Wen and Kaiwei Zhang and Zijian Chen and Fangfang Lu and Xiongkuo Min and Guangtao Zhai and Erjia Xiao and Lingfeng Zhang and Zhenjie Su and Hao Cheng and Yu Liu and Renjing Xu and Long Chen and Xiaoshuai Hao and Jianqin Wu and Jianain Wu and Xuxu Wang Oian Yu and Bo Hu and Weiwei Wang and Pinxin Liu and Yunlong Tang and Luchuan Song and Jinxi He and Jiaru Wu and Hanjia Lyu},
  title = {{VQualA 2025 Challenge on Engagement Prediction for Short Videos: Methods and Results}},
  year = {2025},
  eprint = {2509.02969},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.02969},
}
Figure