Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

Han Liu, Yinwei Wei, Fan Liu, Wenjie Wang, Liqiang Nie, Tat-Seng Chua

Introduction

Micro-videos, such as those on TikTok or Kwai, are inherently multimodal, integrating visual, acoustic, and textual information. Traditional recommendation systems often rely on static multimodal fusion, which assumes a uniform relationship between these modalities across all videos. However, the importance of each modality varies significantly depending on the content. For instance, in a music video, audio and visuals are equally critical, whereas in an educational video, the audio and captions might be redundant while visuals play a secondary role. Static models fail to capture these nuances, leading to sub-optimal recommendations.

Comparison of static fusion, task-based fusion, and dynamic fusion.

The MetaMMF Framework

To address the limitations of static fusion, the authors propose Meta Multimodal Fusion (MetaMMF). This framework reframes multimodal fusion as a meta-learning task, where fusion parameters are dynamically generated for each specific video.

Detailed architecture of the MetaMMF framework, including the Meta Information Extractor and Meta Fusion Learner.

Core Components

  1. Meta Information Extractor: This component uses a Multi-Layer Perceptron (MLP) to compress raw multimodal features (visual, acoustic, textual) into a concise "meta information" vector. This vector captures the specific task-specific needs of the video.
  2. Meta Fusion Learner: Instead of fixed weights, this learner generates fusion network weights on-the-fly. It maps the meta information vector to a video-specific weight matrix using a task-shared 3D tensor.
  3. Efficiency via CP Decomposition: To manage the computational cost of 3D tensors, the authors employ Canonical Polyadic (CP) Decomposition. This factorizes large tensors into smaller matrices, reducing the parameter count to approximately 1% of the original size without sacrificing performance.

MetaMMF is model-agnostic and can be integrated into various recommendation architectures, such as Matrix Factorization (MetaMMF_MF) or Graph Convolutional Networks (MetaMMF_GCN).

Experimental Evaluation

The model was evaluated on three large-scale datasets: MovieLens (augmented with trailers and descriptions), TikTok, and Kwai (which tested the model's robustness to missing acoustic features).

Key Results

  • Performance Gains: MetaMMF_GCN consistently outperformed state-of-the-art baselines like MMGCN and LATTICE. On the NDCG@10 metric, it showed improvements of 5.27% on MovieLens, 6.54% on TikTok, and 4.46% on Kwai over the strongest baselines.
  • Clustering and Efficiency: t-SNE visualizations indicated that MetaMMF produces more distinct user-item interaction clusters. Furthermore, CP Decomposition enabled faster convergence and lower training latency compared to non-decomposed versions.

Conclusion

MetaMMF represents a significant shift from "one-size-fits-all" fusion to a dynamic, item-specific approach. By leveraging meta-learning and efficient tensor decomposition, it provides a scalable and effective solution for multimodal micro-video recommendation. Future work may explore user-side dynamics and more complex fusion functions.

Summary updated

BibTeX
@article{liu2024dynamic,
  author = {Han Liu and Yinwei Wei and Fan Liu and Wenjie Wang and Liqiang Nie and Tat-Seng Chua},
  title = {{Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation}},
  journal = {ACM Trans. Information Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=RpKpwRXEef},
}
Figure