Abstract
Personalization of natural language generation plays a vital role in a large spectrum of tasks, such as explainable recommendation, review summarization and dialog systems. In these tasks, user and item IDs are important identifiers for personalization. Transformer, which is demonstrated with strong language modeling capability, however, is not personalized and fails to make use of the user and item IDs since the ID tokens are not even in the same semantic space as the words. To address this problem, we present a PErsonalized Transformer for Explainable Recommendation (PETER), on which we design a simple and effective learning objective that utilizes the IDs to predict the words in the target explanation, so as to endow the IDs with linguistic meanings and to achieve personalized Transformer. Besides generating explanations, PETER can also make recommendations, which makes it a unified model for the whole recommendation-explanation pipeline. Extensive experiments show that our small unpretrained model outperforms fine-tuned BERT on the generation task, in terms of both effectiveness and efficiency, which highlights the importance and the nice utility of our design.
Introduction
In modern recommendation systems, providing a numerical rating or a list of items is often insufficient; users increasingly expect natural language explanations that justify why a specific product was suggested. While Transformer-based models have become the standard for text generation, they struggle with personalization. The primary challenge lies in the semantic mismatch between discrete user/item IDs and the natural language vocabulary. Because IDs are sparse and lack inherent linguistic meaning, standard Transformers often ignore them, leading to the "identical explanation" problem where the model generates generic, non-personalized text for every user.
Methodology
The researchers propose PETER (PErsonalized Transformer for Explainable Recommendation), a lightweight, unpretrained Transformer model designed to unify recommendation and explanation tasks.
Multi-Task Learning Objectives
PETER is trained on three simultaneous tasks to bridge the gap between IDs and words:
- Context Prediction: This is the model's core innovation. It forces the model to predict which words will appear in the explanation based solely on the user and item IDs. This "drafting" process endows the ID embeddings with linguistic semantics.
- Explanation Generation: The model generates a coherent sentence word-by-word, building upon the context established by the IDs.
- Rating Prediction: An MLP head estimates a numerical rating, allowing the model to function as a complete recommendation engine.
PETER Masking
To support these diverse tasks within a single architecture, the authors developed a custom attention strategy called PETER Masking.
Unlike standard left-to-right causal masking, PETER Masking allows user and item IDs to "see" each other. This ensures that the recommendation and context prediction tasks have access to the full interaction context, while the generation task remains autoregressive to prevent the model from looking at future words during training.
Experiments and Results
The model was evaluated on three major datasets: Yelp (1.2M records), Amazon Movies & TV (441k records), and TripAdvisor (320k records). Each record consists of a user ID, item ID, rating, and a ground-truth explanation extracted from a review.
Key Findings
- Superior Personalization: PETER significantly improved the Unique Sentence Ratio (USR). While standard Transformers often repeat the same common sentences, PETER generates diverse content tailored to specific user-item pairs.
- Efficiency and Performance: Despite having only 2 layers and no pretraining, PETER+ (the version incorporating item features) outperformed fine-tuned BERT-based models (ACMLM) in text quality metrics like BLEU and ROUGE.
- Training Speed: PETER+ is remarkably efficient, training in approximately 57.7 minutes on the TripAdvisor dataset compared to 97.0 minutes for ACMLM, with a per-epoch time of just 2.3 minutes.
Conclusion
PETER demonstrates that architectural design and task-specific objectives—specifically the Context Prediction task and custom masking—can be more effective for personalized recommendation than massive pretraining. By forcing ID embeddings into the same semantic space as natural language, PETER provides a unified, efficient solution for explainable recommendation. Future work will explore extending this framework to multimodal settings and cross-lingual explanation generation.
Summary updated