TL;DR
This paper introduces a personalized micro-video recommendation framework that leverages multimodal subspace coding and an attention-based reordering model. By mapping visual and textual features into a shared latent subspace and using an "interest-related product similarity" module, the system significantly outperforms traditional collaborative filtering and deep learning baselines on sparse datasets.
What Problem Does This Paper Address?
Micro-videos (like those on TikTok or Reels) present a unique "information overload" challenge. They are short, semantically dense, and generated in massive volumes. Traditional recommendation engines often struggle because:
- Search isn't Personal: Keyword-based search returns the same results for everyone, ignoring individual user characteristics.
- Linear Limitations: Simple models like Logistic Regression can't capture high-order, nonlinear interactions between features.
- Data Sparsity: In massive datasets, most users only interact with a tiny fraction of content, making it hard for standard models to find meaningful patterns.
- Multimodal Complexity: Video data contains visual, audio, and textual information that are intrinsically linked, but often treated as isolated silos.
Methods
The authors propose an end-to-end model consisting of three main components:
- Multimodal Subspace Coding Learning: Instead of just concatenating different features (like text and images), the model uses a coding network to map all modalities into a shared potential representation (subspace $H$). This captures both the consistency (what modalities agree on) and complementarity (unique info from each) of the data.
- Feature Filtering (Scene & Behavior): The system uses dual-flow CNNs to extract specific semantic features—namely scene recognition and behavior recognition. This ensures the model retains high-impact probability values for what is actually happening in the video.
- Interest-Related Product Similarity Model: This is the "reordering" stage. It uses an attention mechanism to weigh the importance of a user's historical interactions differently. Finally, it employs a Factorization Machine (FM) approach to model the two-by-two interactions between discrete features, predicting the final preference rate.
Datasets
The model was evaluated on two major benchmarks:
- MovieLens 1M: Since the standard version lacks multimodal data, the authors used the OMDb API to crawl movie posters (visual) and synopses (text) to create a rich multimodal dataset.
- Amazon (Clothing, Shoes & Jewelry): A partial subset including product images and text descriptions. This dataset is significantly sparser than MovieLens, providing a rigorous test for the model's ability to handle "cold-start" or sparse data scenarios.
Key Insights and Ideas
- Subspace > Concatenation: The "aha moment" is that forcing different modalities into a shared subspace acts as a regularizer and a feature extractor simultaneously. It prevents the model from being overwhelmed by the noise of a single modality.
- Knowledge Sharing: The authors found that "knowledge sharing units" (transferring learned features between modalities) perform better than a simple weighted summation of multimodal data.
- Attention Solves Sparsity: On the Amazon dataset—which is two orders of magnitude sparser than MovieLens—the attention-based model showed the most significant gains. This suggests that when data is thin, knowing which past interactions to pay attention to is more important than just having more data.
Key Results
The proposed model (referred to as "Ours" or "Multimodal IRIS" in the text) consistently beat state-of-the-art baselines like Neural Collaborative Filtering (NCF) and Visual Bayesian Personalized Ranking (VBPR).
- MovieLens 1M: Achieved an HR@10 of 0.8287 and NDCG@10 of 0.5899, outperforming the next best model (NAIS) by about 1.6% in ranking quality.
- Amazon Dataset: The lead was even more pronounced here, with an HR@10 of 0.4643 (compared to NAIS at 0.4233) and NDCG@10 of 0.2879 (compared to NAIS at 0.2431).
- Efficiency: By using correlation relationships between video semantics, the model reduces the computational cost of semantic detection while improving search quality.
Limitations and Future Work
While the results are strong, the paper notes a few areas for growth:
- Extreme Sparsity: Even with attention, extremely sparse datasets remain a challenge for learning hidden factors effectively.
- Modal Expansion: The current work focuses heavily on visual and textual data. Integrating audio features or social graph metadata could further refine the "interest-related" module.
- Real-time Constraints: The paper mentions high-speed retrieval as a goal, but more detailed analysis on the latency of the end-to-end reordering model in a production environment would be beneficial.
Summary updated