Abstract
Traditional video recommendation systems often struggle to capture the granular, moment-to-moment preferences of users. This paper introduces the Image-Text Fusion model with Herding Effect Attention (ITF-HEA), a novel framework designed for platforms featuring Time-Sync Comments (TSCs). By modeling the "Herding Effect"—the social phenomenon where users are influenced by preceding comments—and fusing textual data with real-time visual features, ITF-HEA achieves state-of-the-art performance in personalized video recommendation.
Introduction: The Rise of Time-Sync Comments
On video-sharing platforms like Bilibili, Time-Sync Comments (TSCs), or "danmu," have transformed the viewing experience into a social event. Unlike static reviews, TSCs are tied to specific timestamps, offering a real-time window into user sentiment. However, these comments are not independent; they are often influenced by the "Herding Effect," where a user's comment is triggered or shaped by what others have said just moments before.
Existing models typically treat TSCs as isolated text snippets, ignoring the temporal and social context. To address this, we propose a model that accounts for both the visual context of the video and the social dynamics of the comment stream.
Methodology: The ITF-HEA Framework
The proposed ITF-HEA model integrates three sophisticated components to capture the complexity of user interactions:
- Text-based Model (TM): Raw TSCs are processed through a Bidirectional Long Short-Term Memory (Bi-LSTM) network to generate semantic vector representations.
- Image-Text Fusion (ITF): To bridge the "information gap" between text and video, the model extracts visual features from the video frame corresponding to each TSC's timestamp using a pre-trained CNN. These visual features are fused with the text embeddings.
- Herding Effect Attention (HEA): This core innovation uses an encoder-decoder attention mechanism to weigh the influence of previous comments ($M$ preceding TSCs) on the current interaction. The attention weights are calculated based on semantic similarity and a temporal decay function ($e^{-\beta \Delta t}$), reflecting how older comments lose influence over time.
The resulting fused representation is then utilized within a Collaborative Filtering (CF) framework to predict user preferences.
Experimental Evaluation
The model was evaluated using a large-scale dataset from Bilibili, focusing on the "movie" category. The dataset comprised over 1.3 million TSCs across 871 videos. Sentiment analysis was employed to establish ground-truth labels for user "likeness."
Key Results
ITF-HEA was compared against several baseline models, including text-only (HFT) and visual-only (VBPR) systems.
- Performance: ITF-HEA achieved a Precision of 0.976 and an F1-score of 0.3948 for Top-5 recommendations.
- Improvement: The model demonstrated a 3.78% improvement in F1-score over the previous state-of-the-art (KFRCI).
- Ablation Study: Results confirmed that the Herding Effect Attention (HEA) component is a more significant predictor of user preference than visual features alone, highlighting the importance of social context in TSC-based platforms.
Conclusion
The ITF-HEA model successfully demonstrates that social influence and multimodal fusion are critical for accurate video recommendations in the era of interactive media. By mathematically modeling the Herding Effect and its temporal decay, the system captures the evolving "vibe" of a video community, leading to more personalized and context-aware recommendations. Future work will explore more complex fusion architectures and the impact of specific video segments on long-term user profiles.
Summary updated