Introduction
Micro-videos, such as those on TikTok or Kwai, are inherently multimodal, integrating visual, acoustic, and textual information. Traditional recommendation systems often rely on static multimodal fusion, which assumes a uniform relationship between these modalities across all videos. However, the importance of each modality varies significantly depending on the content. For instance, in a music video, audio and visuals are equally critical, whereas in an educational video, the audio and captions might be redundant while visuals play a secondary role. Static models fail to capture these nuances, leading to sub-optimal recommendations.
The MetaMMF Framework
To address the limitations of static fusion, the authors propose Meta Multimodal Fusion (MetaMMF). This framework reframes multimodal fusion as a meta-learning task, where fusion parameters are dynamically generated for each specific video.
Core Components
- Meta Information Extractor: This component uses a Multi-Layer Perceptron (MLP) to compress raw multimodal features (visual, acoustic, textual) into a concise "meta information" vector. This vector captures the specific task-specific needs of the video.
- Meta Fusion Learner: Instead of fixed weights, this learner generates fusion network weights on-the-fly. It maps the meta information vector to a video-specific weight matrix using a task-shared 3D tensor.
- Efficiency via CP Decomposition: To manage the computational cost of 3D tensors, the authors employ Canonical Polyadic (CP) Decomposition. This factorizes large tensors into smaller matrices, reducing the parameter count to approximately 1% of the original size without sacrificing performance.
MetaMMF is model-agnostic and can be integrated into various recommendation architectures, such as Matrix Factorization (MetaMMF_MF) or Graph Convolutional Networks (MetaMMF_GCN).
Experimental Evaluation
The model was evaluated on three large-scale datasets: MovieLens (augmented with trailers and descriptions), TikTok, and Kwai (which tested the model's robustness to missing acoustic features).
Key Results
- Performance Gains: MetaMMF_GCN consistently outperformed state-of-the-art baselines like MMGCN and LATTICE. On the NDCG@10 metric, it showed improvements of 5.27% on MovieLens, 6.54% on TikTok, and 4.46% on Kwai over the strongest baselines.
- Clustering and Efficiency: t-SNE visualizations indicated that MetaMMF produces more distinct user-item interaction clusters. Furthermore, CP Decomposition enabled faster convergence and lower training latency compared to non-decomposed versions.
Conclusion
MetaMMF represents a significant shift from "one-size-fits-all" fusion to a dynamic, item-specific approach. By leveraging meta-learning and efficient tensor decomposition, it provides a scalable and effective solution for multimodal micro-video recommendation. Future work may explore user-side dynamics and more complex fusion functions.
Summary updated