Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

Han Liu, Yinwei Wei, Fan Liu, Wenjie Wang, Liqiang Nie, Tat-Seng Chua

TL;DR

This paper introduces MetaMMF, a framework that moves away from "one-size-fits-all" multimodal fusion in micro-video recommendation. By treating the fusion of visual, acoustic, and textual data for each video as a unique task, the model uses meta-learning to dynamically generate custom fusion parameters, leading to significantly more accurate and personalized recommendations.

What Problem Does This Paper Address?

Micro-videos (like those on TikTok or Kwai) are inherently multimodal, combining sight, sound, and text. Existing recommendation systems typically use static multimodal fusion, assuming that the relationship between these modalities is the same for every video.

The authors argue this is a fundamental flaw. For example, in a music video, the acoustic and visual tracks are complementary and equally vital. In an educational video, the audio and text (captions) might be highly redundant, while the visuals are secondary. Static models "under-fit" these varying relationships. The challenge is to create a system that can adapt its fusion strategy to the specific content of every single video in a massive library.

Methods

The authors propose Meta Multimodal Fusion (MetaMMF), which consists of three primary stages:

  1. Meta Information Extractor: A Multi-Layer Perceptron (MLP) takes the raw multimodal features (visual, acoustic, textual) and compresses them into a concise "meta information" vector. This vector captures the "essence" of the video's task-specific needs.
  2. Meta Fusion Learner: This is the core innovation. Instead of having fixed weights, the fusion network’s weights are generated on-the-fly. The learner uses a task-shared 3D tensor to map the meta information vector into a 2D weight matrix specific to that video.
  3. Model Simplification (CP Decomposition): Using 3D tensors for weight generation is computationally expensive. To keep the model lightweight, the authors apply Canonical Polyadic (CP) Decomposition, factorizing the large tensors into smaller matrices. This reduces the parameter count to nearly 1% of the original size while maintaining performance.

MetaMMF is model-agnostic, meaning it can be plugged into existing recommendation architectures like Matrix Factorization (MetaMMF_MF) or Graph Convolutional Networks (MetaMMF_GCN).

Datasets

The model was evaluated on three large-scale, real-world datasets:

  • MovieLens: Augmented with movie trailers (visual/acoustic) and descriptions (textual).
  • TikTok: A standard micro-video dataset featuring user interactions and multimodal features.
  • Kwai: A dataset from the Kwai platform (notably, this dataset lacked acoustic features, testing the model's ability to handle missing modalities).

Key Insights and Ideas

  • Fusion as a Task: The "aha moment" is reframing multimodal fusion not as a fixed operation, but as an independent meta-learning task for every item.
  • Dynamic vs. Static: The authors demonstrate that even a small amount of dynamic parameterization (fine-tuning a shared weight matrix with a task-specific "delta") is far more effective than a massive, fixed neural network.
  • Efficiency through Math: The use of CP Decomposition proves that "dynamic" doesn't have to mean "slow." By simplifying the 3D tensors, they achieved faster convergence and lower latency than many static SOTA models.
  • The Role of Static Weights: Interestingly, the authors found that keeping a "base" set of static weights (shared across all videos) is still necessary. These act as a "universal prior" that the meta-learner then refines for specific videos.

Key Results

  • State-of-the-Art Performance: MetaMMF_GCN consistently outperformed top-tier baselines like MMGCN, LATTICE, and InvRL.
  • Significant Gains: On the NDCG@10 metric (a measure of ranking quality), MetaMMF_GCN showed improvements of 5.27% (MovieLens), 6.54% (TikTok), and 4.46% (Kwai) over the strongest baselines.
  • Clustering Quality: Visualization using t-SNE showed that MetaMMF created much tighter, more distinct clusters of user-item interactions in the embedding space compared to static models like VBPR.
  • Speedup: Thanks to CP Decomposition, the model converged in fewer epochs and required less training time per epoch than the non-decomposed version.

Limitations and Future Work

  • Cold-Start Scenarios: While the authors mention MetaMMF could help with cold-start problems (recommending new videos with no interaction history), they did not focus heavily on this in the current experiments.
  • Complexity of Fusion Functions: This work focused on neural network-based fusion. The authors suggest that exploring other types of fusion, such as multiple kernel learning or graphical models within a meta-learning framework, could be a fruitful next step.
  • User-Side Dynamics: Currently, the model focuses on the item's multimodal features. Future work could involve dynamically integrating user preferences for different modalities (e.g., some users might care more about audio quality than others).

Summary updated

Abstract

Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this article, we develop a novel meta-learning-based multimodal fusion framework called Meta Multimodal Fusion (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at https://github.com/hanliu95/MetaMMF.
BibTeX
@article{liu2024dynamic,
  author = {Han Liu and Yinwei Wei and Fan Liu and Wenjie Wang and Liqiang Nie and Tat-Seng Chua},
  title = {{Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation}},
  journal = {ACM Trans. Information Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=RpKpwRXEef},
}

Loading related content...

Figure