Abstract
Online reviews in social networks reflect users' preferences for specific attributes of products. Accurate predictions of user ratings based on these reviews can help businesses better understand genuine user feedback. The purpose of this study is to fine-tune large language models using online reviews and corresponding user rating data, generating a large model for predicting user ratings based on reviews. An attention mechanism is introduced to calculate attention weights for fine-grained review texts, reflecting the contribution of different text features to user rating prediction. By visualizing these weights, the process of calculating the predicted rating values can be explained. Experimental results show that the proposed interpretable user rating prediction method can effectively visualize the attention weights of important text features in the decision-making process of the large rating prediction model. Compared to the baseline model, the mean absolute error is reduced by 1.96, and the root mean square error is reduced by 1.73.
Introduction
In the contemporary e-commerce landscape, user ratings serve as a critical metric for product success. However, numerical ratings often lack the context found within the accompanying text reviews. While Large Language Models (LLMs) like BERT or GPT-3 excel at general language understanding, they frequently struggle with domain-specific tasks such as mapping nuanced customer feedback to a standard 1–5 star scale.
This research addresses two primary challenges: the domain gap, where general LLMs are not optimized for the specific sentiment patterns in product reviews, and interpretability, the need to explain the "black box" decision-making process of deep learning models.
Methodology
The proposed technical roadmap involves a two-stage process: supervised fine-tuning and the integration of a custom attention layer.
Supervised Fine-Tuning
Rather than utilizing "off-the-shelf" models, the researchers fine-tuned several prominent LLMs—including BERT, GPT-3, RoBERTa, XLNet, ESM, and T5—on datasets containing paired reviews and ratings. This process enables the models to capture domain-specific linguistic features and sentiment tendencies.
The Attention Mechanism
The core innovation is the introduction of a specific attention mechanism between the LLM’s hidden state and the final output layer. This layer calculates "attention weights" for individual tokens, allowing the model to prioritize specific words (e.g., "mild," "hesitant") that most significantly influence the final rating.
Experimental Setup
The study utilized two massive benchmark datasets for validation:
- Amazon Fine Foods: Comprising 568,454 reviews from 256,059 users across 74,258 products.
- Movies: A larger dataset containing over 7.9 million reviews from approximately 890,000 users.
These datasets provided the ground truth ratings necessary to train and evaluate the accuracy of the fine-tuned models.
Key Results and Discussion
The fine-tuned models demonstrated substantial performance gains over non-fine-tuned baselines. The Mean Absolute Error (MAE) was reduced by 1.96, and the Root Mean Square Error (RMSE) was reduced by 1.73. On a 5-point scale, this reduction represents a shift from vague estimation to high precision.
Convergence and Stability
Experimental data indicated that model performance typically stabilizes after approximately 50 epochs. This convergence suggests a training "sweet spot" where the model effectively captures patterns without overfitting.
Interpretability through Visualization
By ranking words by their attention weights, the researchers provided a clear audit trail for predictions. Tokens with higher weights directly correlated to the sentiment significance of the review, such as flavor intensity in food reviews.
Conclusion
While the proposed method significantly improves rating prediction accuracy and interpretability, challenges remain regarding computational complexity ($O(n^2)$ for the attention mechanism) and potential data bias. Future work will focus on hyperparameter optimization and testing the framework in more niche vertical domains.
Summary updated