Knowledge-aware Collaborative Filtering with Pre-trained Language Model for Personalized Review-based Rating Prediction

Quanxiu Wang, Xinlei Cao, Jianyong Wang, Wei Zhang

Introduction

Personalized recommendation systems increasingly rely on user reviews to capture nuanced preferences. However, effectively integrating unstructured text with structured domain knowledge remains a significant challenge. Existing models often treat reviews as static text or rely on predefined attributes, failing to capture the dynamic relationship between specific product aspects and user sentiment. Furthermore, while Pre-trained Language Models (PLMs) like BERT have set new benchmarks in NLP, their potential in recommendation tasks is often limited by using them as fixed feature extractors rather than fully integrated, fine-tuned components.

To address these gaps, the authors propose KCF-PLM (Knowledge-aware Collaborative Filtering with Pre-trained Language Model), a framework that fuses fine-tuned BERT models with a multi-relational Knowledge Graph (KG) and a dedicated Transformer for aspect modeling.

The KCF-PLM Framework

The KCF-PLM architecture employs a multi-branch approach to synthesize information from various sources:

  • Knowledge Graph & R-GCNs: A heterogeneous KG is constructed featuring users, items, and extracted aspects. Six distinct edge types (e.g., "user cares about aspect," "item is good/bad for aspect") define the relationships. A Relational Graph Convolutional Network (R-GCN) propagates information across this graph to generate structural embeddings.
  • Dual-BERT Branch: Unstructured review text is processed through two BERT instances—U-BERT for users and I-BERT for items. Crucially, these models are fine-tuned end-to-end, allowing them to adapt to the specific linguistic patterns of the recommendation domain.
  • Aspect Transformer: Fine-grained aspects extracted from reviews are processed by a dedicated Transformer. This captures contextual interactions between different attributes mentioned by a user.
  • User-Item Guided Attention: To ensure relevance, the model uses combined user and item representations to guide an attention mechanism, prioritizing aspects most pertinent to a specific interaction.
  • Rating Prediction: Final representations are fused and passed through a Factorization Machine (FM) to predict the rating score.
Detailed architecture of the KCF-PLM framework, showing the integration of BERT, R-GCNs, and the Aspect Transformer.

Experimental Setup and Datasets

The model was evaluated on six large-scale datasets: five categories from Amazon (Clothing, Movies, Toys, Automotive, CellPhones) and Yelp. These datasets provide a diverse range of user-item interactions and review styles. The complexity of the constructed Knowledge Graphs, including the number of nodes and edges, highlights the scale of the structural information integrated into the model.

Dataset and Knowledge Graph statistics across the six evaluation benchmarks.

Key Insights and Performance

The study yielded several critical findings:

  • The Impact of Fine-Tuning: Ablation studies confirmed that fine-tuning BERT is essential. Freezing BERT parameters led to a significant performance drop, indicating that the model must adapt its linguistic understanding to the recommendation context.
  • Aspect Weighting: A frequency-based weighting scheme for aspects proved effective. Aspects mentioned more frequently in reviews are given higher priority, reflecting their central role in the user's evaluation.
  • Node-Type Loss: An auxiliary task—predicting node types (user, item, or aspect)—improved the quality of KG embeddings by helping the R-GCN learn more distinct representations.

In terms of performance, KCF-PLM consistently outperformed strong baselines like DeepCoNN, NARRE, and DAML. It achieved an error reduction (MSE) of up to 6.14% on the Yelp dataset. Despite its complexity, the model maintains an inference time of approximately 7ms per instance, making it viable for real-time applications.

Conclusion and Future Directions

KCF-PLM demonstrates the power of combining deep linguistic features from PLMs with structured knowledge from KGs. While the computational cost of training is high, the gains in accuracy and the feasibility of inference suggest a promising path for next-generation recommendation systems. Future work will focus on model compression techniques, such as knowledge distillation, and exploring the model's effectiveness in cold-start scenarios where review data is sparse.

Summary updated

BibTeX
@article{wang2024knowledge,
  author = {Quanxiu Wang and Xinlei Cao and Jianyong Wang and Wei Zhang},
  title = {{Knowledge-aware Collaborative Filtering with Pre-trained Language Model for Personalized Review-based Rating Prediction}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2024},
  url = {https://openreview.net/forum?id=9eCMLw1xzn},
}
Figure