Abstract
Predicting the popularity of short-form video content is a complex task due to the high variance and multi-modal nature of social media data. This paper presents the winning strategy of Team 25 in the INFORMS 2024 Data Challenge. The approach utilizes a three-pronged pipeline for feature extraction—video, textual, and tabular—and employs an ensemble of neural networks and gradient-boosted trees. By leveraging state-of-the-art video backbones and large language model (LLM) generated descriptions, the team achieved first place on the competition leaderboard, demonstrating the effectiveness of multi-modal fusion in predicting engagement metrics like view, like, comment, and share counts.
Introduction
The ability to predict the popularity of short-form video content is a "holy grail" for content creators and social media platforms alike. However, it is a notoriously difficult task because popularity is driven by a complex, non-linear interplay of visual aesthetics, semantic content, and metadata such as hashtags and posting times. The authors address this challenge by predicting four key engagement metrics: view count, heart (like) count, comment count, and share count. The primary hurdle in this domain is the high variance and heavily skewed distribution of social media data, where a small fraction of "viral" hits exists alongside a vast majority of low-engagement videos.
Methodology
The authors developed a comprehensive three-pronged pipeline to extract features from different modalities before fusing them into a final prediction model.
Video Modality
To capture the visual essence of the content, the team utilized four state-of-the-art (SOTA) backbone networks: TimeSformer, ViViT, VideoMAE, and X-CLIP. These models were used to extract spatiotemporal features directly from the raw video files, capturing motion and visual patterns that correlate with viewer interest.
Textual Modality
A key innovation in this work is the "video-to-text" approach. Rather than relying solely on raw user-provided captions, the team used LLaVA-NeXT and InternVideo2 to generate detailed descriptions. By using custom prompts designed to analyze storytelling, pacing, and emotional tone, they converted subjective visual qualities into structured text. These AI-generated descriptions were then encoded into high-dimensional vectors using BERT.
Tabular Modality
Extensive feature engineering was performed on the associated metadata. Technical specifications such as duration and frame rate were extracted using moviepy. The team also calculated hashtag and mention frequencies and normalized posting times. Logarithmic transformations were applied to handle highly skewed features like author follower counts, ensuring better model stability.
The Ensemble Strategy
The final architecture combined a Neural Network, trained on the fused multi-modal vectors, and XGBoost, trained on the tabular data. The authors observed a systematic bias: the Neural Network tended to be conservative due to Mean Squared Error (MSE) loss optimization on skewed data, while XGBoost was more prone to "exaggerated" or outlier predictions. Averaging the outputs of these two models provided a more robust and accurate final result.
Datasets and Preprocessing
The study utilized the dataset provided by the INFORMS 2024 Data Challenge. The content included short videos and associated tabular metadata (author info, captions, timestamps). A significant preprocessing step involved handling data quality; the team used ffmpeg to re-encode corrupted videos into H.264. Out of the initial set, 1,588 healthy videos were retained for the video-specific feature extraction. The primary evaluation metric for the competition was Mean Absolute Percentage Error (MAPE).
Results and Discussion
The team's multi-modal approach led them to first place on the competition leaderboard. The results demonstrate that the "Final Fusion" model consistently outperformed models using either video or tabular features in isolation.
Key insights from the results include:
- LLMs as Feature Extractors: Converting visual "vibes" into text descriptions allows models to process semantic nuances like "plot conflict" or "scene transitions" that are difficult to capture with standard vision backbones.
- X-CLIP Dominance: Among the video backbones, X-CLIP performed the best, likely due to its design for aligning video and text semantics—a crucial factor for social media content.
- Temporal Importance: Feature importance analysis revealed that normalized creation dates and video duration were among the most critical predictors, highlighting the impact of posting strategy.
Conclusion and Future Work
Despite the success, the study faced limitations, including significant data corruption in the training set and geographic API restrictions that limited real-time trend analysis. Future research will focus on further refining the prompt framework for video-to-text models to yield even deeper semantic features. This is particularly relevant for improving the prediction of "Share" counts, which remain the most difficult metric to forecast accurately.
Summary updated