Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities

Yu Ye, Junchen Fu, Yu Song, Kaiwen Zheng, Joemon M. Jose

Abstract

Multimodal recommendation (MMRec) has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings enhances recommendation performance. However, this assumption lacks comprehensive empirical verification. We conduct a large-scale empirical study examining the role of text and visual embeddings in 14 state-of-the-art MMRec models. Our findings reveal that while multimodal embeddings generally enhance performance in sophisticated graph-based models, the text modality alone often achieves performance comparable to the full multimodal setting, whereas the image modality does not. Furthermore, many common baselines rely primarily on collaborative filtering signals rather than the multimodal data itself.

Introduction

In recent years, the recommendation system (RecSys) community has pivoted toward "multimodal" models that incorporate item descriptions (text) and product photos (images). The logic is intuitive: more information should lead to better user-item matching. However, a critical research gap exists: most studies focus on novel architectures without verifying if the embeddings themselves are providing the performance gains. This study addresses the central question: Are these embeddings truly beneficial, or are we building increasingly complex models that ignore the extra data?

Methodology: Modality Knockout

To isolate the impact of specific modalities without altering model architectures, we introduce a diagnostic methodology called Modality Knockout. This involves replacing informative embeddings with "junk" data:

  1. Visual Knockout (T-Only): Image embeddings are replaced with constant vectors or Gaussian noise.
  2. Textual Knockout (V-Only): Text embeddings are replaced with noise/constants.
  3. Dual Knockout (ID-Only): Both modalities are replaced, leaving only the collaborative filtering (ID-based) signal.

We applied this strategy to 14 state-of-the-art models, ranging from simple fusion schemes (e.g., VBPR, BM3) to complex hybrid graph approaches (e.g., DRAGON, PGL).

Experimental Setup

The study utilizes three standard Amazon benchmarks: Baby, Clothing, and Sports. Features were extracted using ResNet-50 for images and Sentence-BERT (all-MiniLM-L6-v2) for text.

Statistics of the three Amazon benchmark datasets used in the study.

Key Results and Discussion

Our empirical analysis yielded several surprising insights into how modern models utilize multimodal data.

The Dominance of Textual Information

In almost every model tested, removing text caused a significant performance crash, while removing images often had a negligible effect. The rich semantic information in titles and descriptions appears to be the primary driver of multimodal gains. This "visual blindness" may stem from the fact that image embeddings are more dispersed in the vector space, making them harder for models to learn from compared to text.

The Baseline Illusion

Popular baselines like VBPR and BM3 showed surprising stability during Dual Knockout. While this might seem positive, it indicates a red flag: these models are not effectively utilizing the multimodal data and are essentially functioning as standard ID-based matrix factorization models. In contrast, sophisticated graph-based models (like FREEDOM and DRAGON) saw NDCG@10 scores plummet by 20% to over 60% during Dual Knockout, proving their architectures successfully extract value from the embeddings.

Performance comparison across 14 models on three datasets under different modality knockout scenarios.

Encoder Impact

Switching from ResNet to Vision Transformer (ViT) did not provide a universal boost. While some models like GRCN improved, others like DRAGON performed worse, suggesting that visual utility is highly dependent on how a specific architecture integrates features.

Conclusion

Our study demonstrates that while multimodal embeddings are beneficial, their utility is unevenly distributed. Text is currently the dominant modality, and many simple fusion models fail to leverage multimodal signals effectively. Future work should focus on end-to-end training to help models "see" images better and explore domains where visual data might play a more critical role than in standard e-commerce datasets.

Summary updated

Abstract

Multimodal recommendation (MMRec) has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT, Vision Transformers, and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings can enhance recommendation performance. However, despite its popularity, this assumption lacks comprehensive empirical verification. This presents a critical research gap. To address it, we pose the central research question of this paper: Are multimodal embeddings truly beneficial for recommendation? To answer this question, we conduct a large-scale empirical study examining the role of text and visual embeddings in modern MMRec models, both as a whole and individually. Specifically, we pose two key research questions: (1) Do multimodal embeddings as a whole improve recommendation performance? (2) Is each individual modality - text and image - useful when used alone? To isolate the effect of individual modalities - text or visual - we employ a modality knockout strategy by setting the corresponding embeddings to either constant values or random noise. To ensure the scale and comprehensiveness of our study, we evaluate 14 widely used state-of-the-art MMRec models. Our findings reveal that: (1) multimodal embeddings generally enhance recommendation performance - particularly when integrated through more sophisticated graph-based fusion models. Surprisingly, commonly adopted baseline models with simple fusion schemes, such as VBPR and BM3, show only limited gains. (2) The text modality alone achieves performance comparable to the full multimodal setting in most cases, whereas the image modality alone does not. These results offer foundational insights and practical guidance for the MMRec community. We will release our code and datasets to facilitate future research.
BibTeX
@inproceedings{ye2026are,
  author = {Yu Ye and Junchen Fu and Yu Song and Kaiwen Zheng and Joemon M. Jose},
  title = {{Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities}},
  booktitle = {ECIR},
  year = {2026},
  doi = {10.1007/978-3-032-21324-2\_5},
}
Figure