TL;DR
This paper rigorously tests the assumption that adding text and image embeddings automatically improves recommendation systems. By using a "modality knockout" strategy—replacing embeddings with noise or constant values—the authors reveal that while multimodal data helps sophisticated graph-based models, many popular baselines rely almost entirely on user-item ID interactions, and the textual modality consistently outperforms the visual one.
What Problem Does This Paper Address?
In recent years, the recommendation system (RecSys) community has pivoted toward "multimodal" models that incorporate item descriptions (text) and product photos (images). The logic is intuitive: more information should lead to better user-item matching.
However, the authors identify a critical research gap: most papers focus on novel architectures or fusion strategies without verifying if the embeddings themselves are actually providing the performance gains. Because evaluation protocols vary wildly across studies, it’s often unclear if a model is winning because of its clever graph neural network or simply because it was tuned better on ID-based signals. The central question is: Are these embeddings truly beneficial, or are we just building increasingly complex "black boxes" that ignore the extra data?
Methods
To isolate the impact of specific modalities, the authors introduce a diagnostic methodology called Modality Knockout. Instead of removing a modality entirely (which would change the model's architecture), they keep the architecture intact but replace the informative embeddings with "junk" data:
- Visual Knockout (T-Only): Replace image embeddings with constant vectors or Gaussian noise.
- Textual Knockout (V-Only): Replace text embeddings with noise/constants.
- Dual Knockout (ID-Only): Replace both, leaving only the collaborative filtering (ID-based) signal.
They applied this strategy to 14 state-of-the-art models, categorized into four groups:
- Simple Fusion: (e.g., VBPR, BM3)
- User-Item Graph Integration: (e.g., MMGCN, GRCN)
- Item-Item Graph Construction: (e.g., LATTICE, FREEDOM)
- Hybrid Graph Approaches: (e.g., DRAGON, PGL)
Datasets
The study utilizes three standard Amazon benchmarks:
- Baby
- Clothing
- Sports
For features, they used standard pre-extracted vectors: ResNet-50 for images and all-MiniLM-L6-v2 (Sentence-BERT) for text. They also conducted a case study replacing ResNet-50 with Vision Transformer (ViT) to see if encoder choice changed the outcome.
Key Insights and Ideas
- The Dominance of Text: In almost every model, removing text caused a massive performance crash, while removing images had a negligible effect. The rich semantic information in titles and descriptions is the primary driver of multimodal gains.
- The "Visual Blindness" of Models: Most models are not effectively "seeing" the images. The authors found that image embeddings are much more dispersed in the vector space than text embeddings, making them significantly harder for models to learn from during training.
- The Baseline Illusion: Popular baselines like VBPR and BM3 showed "surprising stability" during Dual Knockout. This sounds good, but it’s actually a red flag: it means these models aren't really using the multimodal data at all. They are essentially functioning as standard ID-based matrix factorization models.
- Sophistication Pays Off: Unlike the simple baselines, graph-based models (like FREEDOM and DRAGON) actually do leverage the embeddings. When those embeddings were knocked out, their performance plummeted, proving that their complex architectures are successfully extracting value from the data.
Key Results
- Performance Drops: In the Dual Knockout scenario, sophisticated models saw NDCG@10 scores plummet by 20% to over 60%. For example, on the Baby dataset, DRAGON's performance dropped by 43%.
- Text vs. Image: For FREEDOM on the Baby dataset, knocking out text dropped NDCG@10 by 28%, whereas knocking out images had a "minimal effect."
- Encoder Impact: Switching from ResNet to ViT didn't provide a universal boost. While some models like GRCN improved (+1.7% to +3.2%), others like DRAGON actually performed worse (-11.1%), suggesting that visual utility is highly dependent on how the specific architecture integrates the features.
Limitations and Future Work
The authors acknowledge that their study relies on pre-extracted, frozen embeddings. It is possible that end-to-end training (where the image/text encoders are updated alongside the recommendation model) could help models "see" images better.
Additionally, the "textual dominance" might be a byproduct of the datasets used (Amazon product data). Future work could explore domains where images are arguably more important than text, such as short-video recommendations or fashion-specific platforms, to see if the visual modality remains underutilized. Finally, the authors plan to use these insights to design new fusion strategies that specifically address the difficulty of learning from dispersed visual embeddings.
Summary updated