Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities

Yu Ye, Junchen Fu, Yu Song, Kaiwen Zheng, Joemon M. Jose

TL;DR

This paper rigorously tests the assumption that adding text and image embeddings automatically improves recommendation systems. By using a "modality knockout" strategy—replacing embeddings with noise or constant values—the authors reveal that while multimodal data helps sophisticated graph-based models, many popular baselines rely almost entirely on user-item ID interactions, and the textual modality consistently outperforms the visual one.

What Problem Does This Paper Address?

In recent years, the recommendation system (RecSys) community has pivoted toward "multimodal" models that incorporate item descriptions (text) and product photos (images). The logic is intuitive: more information should lead to better user-item matching.

However, the authors identify a critical research gap: most papers focus on novel architectures or fusion strategies without verifying if the embeddings themselves are actually providing the performance gains. Because evaluation protocols vary wildly across studies, it’s often unclear if a model is winning because of its clever graph neural network or simply because it was tuned better on ID-based signals. The central question is: Are these embeddings truly beneficial, or are we just building increasingly complex "black boxes" that ignore the extra data?

Methods

To isolate the impact of specific modalities, the authors introduce a diagnostic methodology called Modality Knockout. Instead of removing a modality entirely (which would change the model's architecture), they keep the architecture intact but replace the informative embeddings with "junk" data:

  1. Visual Knockout (T-Only): Replace image embeddings with constant vectors or Gaussian noise.
  2. Textual Knockout (V-Only): Replace text embeddings with noise/constants.
  3. Dual Knockout (ID-Only): Replace both, leaving only the collaborative filtering (ID-based) signal.

They applied this strategy to 14 state-of-the-art models, categorized into four groups:

  • Simple Fusion: (e.g., VBPR, BM3)
  • User-Item Graph Integration: (e.g., MMGCN, GRCN)
  • Item-Item Graph Construction: (e.g., LATTICE, FREEDOM)
  • Hybrid Graph Approaches: (e.g., DRAGON, PGL)

Datasets

The study utilizes three standard Amazon benchmarks:

  • Baby
  • Clothing
  • Sports

For features, they used standard pre-extracted vectors: ResNet-50 for images and all-MiniLM-L6-v2 (Sentence-BERT) for text. They also conducted a case study replacing ResNet-50 with Vision Transformer (ViT) to see if encoder choice changed the outcome.

Key Insights and Ideas

  • The Dominance of Text: In almost every model, removing text caused a massive performance crash, while removing images had a negligible effect. The rich semantic information in titles and descriptions is the primary driver of multimodal gains.
  • The "Visual Blindness" of Models: Most models are not effectively "seeing" the images. The authors found that image embeddings are much more dispersed in the vector space than text embeddings, making them significantly harder for models to learn from during training.
  • The Baseline Illusion: Popular baselines like VBPR and BM3 showed "surprising stability" during Dual Knockout. This sounds good, but it’s actually a red flag: it means these models aren't really using the multimodal data at all. They are essentially functioning as standard ID-based matrix factorization models.
  • Sophistication Pays Off: Unlike the simple baselines, graph-based models (like FREEDOM and DRAGON) actually do leverage the embeddings. When those embeddings were knocked out, their performance plummeted, proving that their complex architectures are successfully extracting value from the data.

Key Results

  • Performance Drops: In the Dual Knockout scenario, sophisticated models saw NDCG@10 scores plummet by 20% to over 60%. For example, on the Baby dataset, DRAGON's performance dropped by 43%.
  • Text vs. Image: For FREEDOM on the Baby dataset, knocking out text dropped NDCG@10 by 28%, whereas knocking out images had a "minimal effect."
  • Encoder Impact: Switching from ResNet to ViT didn't provide a universal boost. While some models like GRCN improved (+1.7% to +3.2%), others like DRAGON actually performed worse (-11.1%), suggesting that visual utility is highly dependent on how the specific architecture integrates the features.

Limitations and Future Work

The authors acknowledge that their study relies on pre-extracted, frozen embeddings. It is possible that end-to-end training (where the image/text encoders are updated alongside the recommendation model) could help models "see" images better.

Additionally, the "textual dominance" might be a byproduct of the datasets used (Amazon product data). Future work could explore domains where images are arguably more important than text, such as short-video recommendations or fashion-specific platforms, to see if the visual modality remains underutilized. Finally, the authors plan to use these insights to design new fusion strategies that specifically address the difficulty of learning from dispersed visual embeddings.

Summary updated

Abstract

Multimodal recommendation has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT, Vision Transformers, and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings can enhance recommendation performance. However, despite its popularity, this assumption lacks comprehensive empirical verification. This presents a critical research gap. To address it, we pose the central research question of this paper: Are multimodal embeddings truly beneficial for recommendation? To answer this question, we conduct a large-scale empirical study examining the role of text and visual embeddings in modern multimodal recommendation models, both as a whole and individually. Specifically, we pose two key research questions: (1) Do multimodal embeddings as a whole improve recommendation performance? (2) Is each individual modality - text and image - useful when used alone? To isolate the effect of individual modalities - text or visual - we employ a modality knockout strategy by setting the corresponding embeddings to either constant values or random noise. To ensure the scale and comprehensiveness of our study, we evaluate 14 widely used state-of-the-art multimodal recommendation models. Our findings reveal that: (1) multimodal embeddings generally enhance recommendation performance - particularly when integrated through more sophisticated graph-based fusion models. Surprisingly, commonly adopted baseline models with simple fusion schemes, such as VBPR and BM3, show only limited gains. (2) The text modality alone achieves performance comparable to the full multimodal setting in most cases, whereas the image modality alone does not. These results offer foundational insights and practical guidance for the multimodal recommendation community.
BibTeX
@inproceedings{ye2026are,
  author = {Yu Ye and Junchen Fu and Yu Song and Kaiwen Zheng and Joemon M. Jose},
  title = {{Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities}},
  booktitle = {ECIR},
  year = {2026},
  doi = {10.1007/978-3-032-21324-2\_5},
}

Loading related content...

Figure