Attentional factorization machine with review-based user–item interaction for recommendation

Zheng Li, Di Jin, Ke YuanScientific 2023 · DOI

In the world of recommender systems, a five-star rating tells you that a user liked a product, but the text of their review tells you why. While researchers have long tried to mine these reviews to improve recommendations, they often hit a wall: human language is messy, context-dependent, and full of noise.

A new paper published in Scientific Reports introduces AFMRUI (Attentional Factorization Machine with Review-based User–Item Interaction), a model that combines state-of-the-art NLP with advanced interaction modeling to bridge the gap between raw text and precise rating prediction.

TL;DR

AFMRUI leverages the power of RoBERTa and BiGRUs to extract deep semantic meaning from user reviews, then employs an Attentional Factorization Machine (AFM) to intelligently weight user-item feature interactions. By distinguishing between useful signals and "noisy" interactions, the model achieves state-of-the-art accuracy across multiple real-world datasets.

What Problem Does This Paper Address?

Most recommendation engines suffer from "data sparsity"—there are millions of products, but any single user only rates a handful. Reviews help fill this gap, but existing review-based models have three major flaws:

  1. Static Semantics: Older models use static word vectors (like Word2Vec or GloVe). These can’t handle polysemy; the word "cricket" means something very different in a sports review than in a pet store review.
  2. Short-Sighted Architectures: Many models use Convolutional Neural Networks (CNNs) to process text. While fast, CNNs struggle to capture long-range dependencies in lengthy, rambling reviews.
  3. Interaction Blindness: When a model combines a "user profile" and an "item profile," it often uses a simple dot product or standard Factorization Machine (FM). This assumes every feature interaction is equally important, whereas, in reality, many interactions are irrelevant noise that degrades prediction quality.

Methods

The authors propose a four-layer architecture to solve these issues:

  1. Review Embedding Layer (RoBERTa): Instead of static vectors, they use RoBERTa to generate dynamic, context-aware embeddings. This ensures the model understands the specific nuance of words based on the surrounding text.
  2. Sequence Encoding Layer (BiGRU): To capture the "flow" of a review, they use Bidirectional Gated Recurrent Units. This allows the model to understand the relationship between words at the beginning and end of a long sentence.
  3. Review-Level Attention: Not all reviews are equally helpful. This layer adaptively weights reviews, focusing on those that contain the most "informative" content for a specific user or item.
  4. Rating Prediction Layer (AFM): This is the core innovation. Instead of just mashing user and item features together, the Attentional Factorization Machine learns which specific feature interactions (e.g., a user’s preference for "bass" interacting with a headphone’s "audio quality" feature) actually drive the rating. It effectively "mutes" the noise from useless interactions.

Datasets

The model was evaluated on five diverse, real-world benchmarks:

  • Amazon Reviews: Four categories including Digital Music, Baby, Office Products, and Beauty.
  • Yelp: A massive dataset of restaurant and service reviews. These datasets are standard in the field because they contain both explicit ratings (1-5 stars) and rich natural language text.

Key Insights and Ideas

  • Context is King: The ablation studies showed that replacing GloVe with RoBERTa provided the single biggest jump in performance. Understanding context is more important for recommendation than simply knowing the vocabulary.
  • The "Noise" of Interaction: The authors found that standard Factorization Machines (FM) often get distracted by irrelevant feature pairings. By adding an attention mechanism to the FM (making it an AFM), the model focuses on the 0.5–0.9 attention score range for meaningful interactions, while ignoring the low-score "noise."
  • BiGRU > CNN: For recommendation tasks, the sequential order of words in a review matters more than the local "n-gram" patterns that CNNs typically pick up.

Key Results

AFMRUI was tested against ten baseline models, including heavy hitters like DeepCoNN, NARRE, and DAML.

  • Accuracy: AFMRUI consistently achieved the lowest Mean Square Error (MSE) and Mean Absolute Error (MAE) across all five datasets.
  • Digital Music Gains: On the Digital Music dataset, AFMRUI improved MSE by approximately 3.7% and MAE by 2.1% over the previous best-performing baseline (MPRS).
  • Scalability: While the model has significantly more parameters (~127 million) due to the RoBERTa backbone, the authors argue the trade-off is worth it for the substantial gain in recommendation quality.

Limitations and Future Work

The authors identify a few areas for growth:

  • Text-Only Focus: Currently, the model relies entirely on review text and ratings. It does not yet incorporate User-Item Interaction Graphs, which could provide structural data about social connections or browsing history.
  • Computational Overhead: Using RoBERTa makes the model "heavy." Future iterations might explore distilled versions of BERT to reduce inference latency for real-time production environments.
  • Cold Start: Like most review-based models, it still requires at least some historical text to function effectively for a new user.

Reference: Li, Z., Jin, D. & Yuan, K. Attentional factorization machine with review-based user–item interaction for recommendation. Sci Rep 13, 13454 (2023). https://doi.org/10.1038/s41598-023-40633-4

Summary updated

Abstract

In recommender systems, user reviews on items contain rich semantic information, which can express users' preferences and item features. However, existing review-based recommendation methods either use the static word vector model or cannot effectively extract long sequence features in reviews, resulting in the limited ability of user feature expression. Furthermore, the impact of different or useless feature interactions between users and items on recommendation performance is ignored. Therefore, we propose an attentional factorization machine with review-based user-item interaction for recommendation (AFMRUI), which first leverages RoBERTa to obtain the embedding feature of each user/item review, and combines bidirectional gated recurrent units with attention network to highlight more useful information in both user and item reviews. Then we adopt AFM to learn user-item feature interactions to distinguish the importance of different user-item feature interactions and further to obtain more accurate rating prediction, so as to promote recommendation. Finally, we conducted performance evaluation on five real-world datasets. Experimental results on five datasets demonstrated that the proposed AFMRUI outperformed the state-of-the-art review-based methods regarding two commonly used evaluation metrics.
BibTeX
@inproceedings{li2023attentional,
  author = {Zheng Li and Di Jin and Ke Yuan},
  title = {{Attentional factorization machine with review-based user–item interaction for recommendation}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-40633-4},
}
Figure