Abstract
Multimodal recommendation (MMRec) has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings enhances recommendation performance. However, this assumption lacks comprehensive empirical verification. We conduct a large-scale empirical study examining the role of text and visual embeddings in 14 state-of-the-art MMRec models. Our findings reveal that while multimodal embeddings generally enhance performance in sophisticated graph-based models, the text modality alone often achieves performance comparable to the full multimodal setting, whereas the image modality does not. Furthermore, many common baselines rely primarily on collaborative filtering signals rather than the multimodal data itself.
Introduction
In recent years, the recommendation system (RecSys) community has pivoted toward "multimodal" models that incorporate item descriptions (text) and product photos (images). The logic is intuitive: more information should lead to better user-item matching. However, a critical research gap exists: most studies focus on novel architectures without verifying if the embeddings themselves are providing the performance gains. This study addresses the central question: Are these embeddings truly beneficial, or are we building increasingly complex models that ignore the extra data?
Methodology: Modality Knockout
To isolate the impact of specific modalities without altering model architectures, we introduce a diagnostic methodology called Modality Knockout. This involves replacing informative embeddings with "junk" data:
- Visual Knockout (T-Only): Image embeddings are replaced with constant vectors or Gaussian noise.
- Textual Knockout (V-Only): Text embeddings are replaced with noise/constants.
- Dual Knockout (ID-Only): Both modalities are replaced, leaving only the collaborative filtering (ID-based) signal.
We applied this strategy to 14 state-of-the-art models, ranging from simple fusion schemes (e.g., VBPR, BM3) to complex hybrid graph approaches (e.g., DRAGON, PGL).
Experimental Setup
The study utilizes three standard Amazon benchmarks: Baby, Clothing, and Sports. Features were extracted using ResNet-50 for images and Sentence-BERT (all-MiniLM-L6-v2) for text.
Key Results and Discussion
Our empirical analysis yielded several surprising insights into how modern models utilize multimodal data.
The Dominance of Textual Information
In almost every model tested, removing text caused a significant performance crash, while removing images often had a negligible effect. The rich semantic information in titles and descriptions appears to be the primary driver of multimodal gains. This "visual blindness" may stem from the fact that image embeddings are more dispersed in the vector space, making them harder for models to learn from compared to text.
The Baseline Illusion
Popular baselines like VBPR and BM3 showed surprising stability during Dual Knockout. While this might seem positive, it indicates a red flag: these models are not effectively utilizing the multimodal data and are essentially functioning as standard ID-based matrix factorization models. In contrast, sophisticated graph-based models (like FREEDOM and DRAGON) saw NDCG@10 scores plummet by 20% to over 60% during Dual Knockout, proving their architectures successfully extract value from the embeddings.
Encoder Impact
Switching from ResNet to Vision Transformer (ViT) did not provide a universal boost. While some models like GRCN improved, others like DRAGON performed worse, suggesting that visual utility is highly dependent on how a specific architecture integrates features.
Conclusion
Our study demonstrates that while multimodal embeddings are beneficial, their utility is unevenly distributed. Text is currently the dominant modality, and many simple fusion models fail to leverage multimodal signals effectively. Future work should focus on end-to-end training to help models "see" images better and explore domains where visual data might play a more critical role than in standard e-commerce datasets.
Summary updated