Personalized Transformer for Explainable Recommendation

Lei Li, Yongfeng Zhang, Li Chen

Abstract

Explainable recommendation systems aim to provide users with both accurate item suggestions and natural language justifications. However, standard Transformer architectures often struggle to bridge the gap between discrete user/item IDs and linguistic tokens, frequently resulting in generic, non-personalized explanations. This paper introduces PETER (PErsonalized Transformer for Explainable Recommendation), a unified framework that utilizes a novel context prediction task and a specialized masking mechanism. PETER allows a compact, unpretrained model to achieve superior performance in both recommendation accuracy and explanation quality, significantly outperforming fine-tuned large-scale models like BERT while maintaining high training efficiency.

1. Introduction

In modern recommender systems, explainability is crucial for building user trust and improving the overall user experience. Providing a personalized explanation, such as "the style of this jacket is fashionable," helps users understand the rationale behind a recommendation. Despite the success of Transformers in natural language processing, their application to explainable recommendation faces two primary hurdles:

  1. The ID Gap: User and Item IDs, which are essential for personalization, reside in a different semantic space than natural language tokens.
  2. The Frequency Problem: Because IDs appear much less frequently than common words, models often ignore these "rare" tokens, relying instead on generic start tokens (e.g., <bos>). This leads to the generation of "identical explanations" that lack user-specific relevance.

2. Methodology

The proposed PETER framework employs a 2-layer Transformer architecture designed for multi-task learning. It simultaneously addresses rating prediction, context prediction, and explanation generation.

The PETER architecture for multi-task learning, including rating prediction, context prediction, and explanation generation.

2.1 PETER Masking

Standard Transformers typically use a strict left-to-right causal mask. In contrast, PETER introduces a specialized masking matrix that allows User ($u$) and Item ($i$) ID tokens to attend to one another. This bidirectional interaction ensures that the initial representations capture the specific user-item relationship before the text generation process begins.

The PETER masking matrix, illustrating how user and item IDs interact and how the sequence is constrained during generation.

2.2 Context Prediction Task

A key innovation of PETER is the "context prediction" task. The model is trained to predict the set of words (features) likely to appear in the final explanation based solely on the User and Item IDs. This task acts as a "drafting" phase, forcing the model to align ID embeddings with linguistic concepts, thereby bridging the semantic gap.

2.3 Multi-Task Learning

The model optimizes a joint loss function comprising three components:

  • Recommendation: Predicting the numerical rating using Mean Squared Error (MSE).
  • Context Prediction: Identifying the relevant content tokens using Negative Log-Likelihood (NLL).
  • Explanation Generation: Sequencing tokens into a coherent sentence using NLL.

3. Experimental Evaluation

PETER was evaluated on three large-scale datasets: Yelp (~1.3M records), Amazon Movies & TV (~441k records), and TripAdvisor Hotels (~320k records). Each dataset provides user-item interactions, ratings, and ground-truth explanations extracted from reviews.

4. Results and Discussion

Experimental results demonstrate PETER's effectiveness across several dimensions:

  • Text Quality: PETER+ (the feature-enhanced version) significantly outperformed baselines, including fine-tuned BERT (ACMLM). On the Yelp dataset, PETER+ achieved a BLEU-4 score of 3.43, compared to just 0.24 for BERT.
  • Personalization: PETER achieved a high Unique Sentence Ratio (USR), indicating its ability to generate diverse, user-specific explanations rather than generic templates.
  • Efficiency: Due to its compact 2-layer architecture, PETER+ is highly efficient. On TripAdvisor, it trained 14 times faster than BERT-based models (2.3 minutes vs. 32.3 minutes per epoch).
  • Recommendation Accuracy: The model maintained recommendation performance comparable to established algorithms like SVD++ and PMF, showing that the explanation task does not compromise the primary goal of accurate prediction.

5. Conclusion

PETER demonstrates that task-specific architecture design can be more effective than massive general-purpose pre-training for specialized tasks like personalized recommendation. By treating IDs as "out-of-vocabulary" tokens and using context prediction to force alignment, PETER generates high-quality, personalized explanations efficiently. Future work includes exploring multimodal explanations and extending the framework to cross-lingual recommendation scenarios.

Summary updated

Abstract

Personalization of natural language generation plays a vital role in a large spectrum of tasks, such as explainable recommendation, review summarization and dialog systems. In these tasks, user and item IDs are important identifiers for personalization. Transformer, which is demonstrated with strong language modeling capability, however, is not personalized and fails to make use of the user and item IDs since the ID tokens are not even in the same semantic space as the words. To address this problem, we present a PErsonalized Transformer for Explainable Recommendation (PETER), on which we design a simple and effective learning objective that utilizes the IDs to predict the words in the target explanation, so as to endow the IDs with linguistic meanings and to achieve personalized Transformer. Besides generating explanations, PETER can also make recommendations, which makes it a unified model for the whole recommendation-explanation pipeline. Extensive experiments show that our small unpretrained model outperforms fine-tuned BERT on the generation task, in terms of both effectiveness and efficiency, which highlights the importance and the nice utility of our design.
BibTeX
@inproceedings{li2021personalized,
  author = {Lei Li and Yongfeng Zhang and Li Chen},
  title = {{Personalized Transformer for Explainable Recommendation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.383},
}
Figure