MMREC: LLM Based Multi-Modal Recommender System

Jiahao Tian, Zhenkai Wang, Jinman Zhao, Zhicheng Ding

Abstract

Modern recommender systems (RS) face significant challenges in processing unstructured, multi-modal data, specifically regarding signal noise and the mathematical misalignment between different data types. This paper introduces MMREC, a novel framework that leverages the reasoning and summarization capabilities of Large Language Models (LLMs) to transform messy user-generated content—such as restaurant reviews and smartphone photos—into a unified, noise-free latent space. By utilizing GPT-3.5 for textual summarization and BLIP-2 for visual-to-textual translation, MMREC achieves a 19.4% improvement in reducing false positive recommendations compared to traditional deep learning baselines. Our findings suggest that LLMs can serve as an effective "denoiser" and bridge for multi-modal alignment in high-trust recommendation environments.

Introduction

As user-generated content becomes the backbone of modern recommendation engines, the ability to effectively synthesize multi-modal information is paramount. However, traditional deep learning architectures often struggle with two primary issues: the "noise" inherent in raw user data and the "modality gap" between text and images. Simply averaging embeddings from dozens of disparate reviews often dilutes the relevant signal, making it difficult for models to distinguish core user preferences. Furthermore, aligning a visual representation of a meal with a textual description of its quality requires complex cross-modal architectures.

MMREC addresses these challenges by employing LLMs to distill essential features and project diverse data types into a single, cohesive latent space. By treating images as "visual text" and summarizing long histories into concise profiles, the system prioritizes clarity and precision over raw data volume.

Methodology

The MMREC framework is built upon the foundational Deep Learning Recommendation Model (DLRM), enhanced with several LLM-driven layers designed for feature extraction and denoising.

LLM-Driven Summarization

Rather than processing raw, unstructured review histories, MMREC utilizes GPT-3.5 to summarize a user’s entire interaction history into a concise profile. This process acts as a sophisticated denoiser, filtering out irrelevant "fluff" and retaining only the most consistent user patterns and preferences.

Unified Latent Space via BLIP-2

To bridge the modality gap, MMREC employs BLIP-2, a vision-language model, to generate descriptive text captions for user-uploaded images. By converting a photo into a textual description (e.g., "a steak sandwich on a cutting board"), the system can process visual data using the same text encoder applied to reviews. This ensures that both visual and textual information reside in the same mathematical space without the need for complex alignment layers.

Feature Engineering and Dimension Reduction

Beyond summarization, the LLM extracts specific dense features (such as price levels) and sparse features (such as restaurant categories) from unstructured text. To prevent the final ranking model from overfitting on high-dimensional data, a Multi-Layer Perceptron (MLP) is used to compress these embeddings into a compact 32-dimensional representation before they reach the final ranking stage.

Experimental Setup

The researchers evaluated MMREC using a comprehensive dataset of Google restaurant reviews. This dataset is characterized by its unstructured nature and diverse imagery, reflecting the imbalanced data common in real-world applications where positive interactions typically outnumber negative ones. The performance of MMREC was benchmarked against a standard DLRM baseline utilizing traditional embedding averaging and ResNet-50 for image processing.

Results and Discussion

The experimental results highlight a significant shift in recommendation performance when utilizing LLM-enhanced features:

  • False Positive Reduction: The optimized MMREC configuration achieved a 19.4% improvement in the False Positive Rate (FPR) over the baseline. This suggests that the LLM's ability to "forget" irrelevant details helps the model avoid recommending items that do not align with the user's core tastes.
  • The Accuracy Trade-off: The reduction in false positives was accompanied by a 10% decrease in overall accuracy. However, in the context of recommendation systems, a lower FPR is often more valuable for building user trust, as recommending a disliked item is generally more detrimental to the user experience than missing a potential interest.
  • Ablation Analysis: Ablation studies confirmed that removing either the LLM-processed text or the image-to-caption descriptions led to a significant drop in performance, validating the synergy between the two modalities.

Conclusion

MMREC demonstrates that LLMs can significantly enhance recommender systems by acting as a bridge between disparate data modalities and as a filter for high-noise environments. While the computational overhead of models like GPT-3.5 and BLIP-2 remains a consideration for real-time deployment, the substantial improvement in recommendation precision offers a promising path forward for creating more reliable and trustworthy AI-driven discovery tools. Future work will focus on optimizing the balance between denoising and information loss to recover raw accuracy while maintaining low false positive rates.

Summary updated

BibTeX
@inproceedings{tian2024mmrec,
  author = {Jiahao Tian and Zhenkai Wang and Jinman Zhao and Zhicheng Ding},
  title = {{MMREC: LLM Based Multi-Modal Recommender System}},
  booktitle = {SMAP},
  year = {2024},
  doi = {10.1109/smap63474.2024.00028},
}
Figure