TransNets for Review Generation

Rose Catherine, William W. CohenWorkshops @ ICLR 2018 · OpenReview

Introduction

In the domain of recommender systems, a simple star rating often lacks the explanatory power needed to justify why a system believes a user will like a specific product. Natural language reviews—written from the perspective of a specific user for a specific item—serve as an ideal, personalized explanation. However, generating these reviews is technically challenging. Standard Recurrent Neural Networks (RNNs) model language well, but steering them to incorporate specific user preferences and item features typically involves simply concatenating user and item IDs to the input at each timestep. This approach often yields generic text or suffers from high perplexity.

To bridge the gap between simple ID-based generation and the complexity of human-written reviews, researchers introduce TransNets, a student-teacher architecture designed to rapidly generate personalized, high-quality reviews.

The TransNet Architecture

The core innovation of TransNets is splitting the generation task into two distinct networks: a Target network (the teacher) and a Source network (the student).

TransNet Architecture
  1. The Target Network (Teacher): During training, this network functions as an autoencoder. It processes an actual review ($rev_{AB}$) through a Convolutional Neural Network (CNN) to create a dense latent representation. An LSTM decoder then attempts to reconstruct the original text from this representation. By processing the real review, the Target network learns an optimal internal representation of the text.
  2. The Source Network (Student): The Source network does not have access to the actual review text. Instead, it only receives the user_A and item_B IDs. It embeds these IDs into a latent space and passes them through a TRANSFORM layer (a deep, non-linear feed-forward network).
  3. The Learning Objective: The model is trained by minimizing the $L_2$ loss between the Source network's output and the Target network's latent representation. Essentially, the student is trained to predict the internal state the teacher would have generated had it seen the real review.

At inference time, the Target network is discarded. The Source network predicts the latent vector based solely on the user and item IDs, and this vector is fed into the pre-trained LSTM decoder to generate the review. This latent mapping proves significantly more effective than traditional concatenation methods.

Experimental Setup

The authors evaluated TransNets using a dense subset of the BeerAdvocate dataset, termed BA_150. This "150-core" subset ensures every user and item has at least 150 associated reviews, resulting in approximately 1,600 users, 1,600 items, and over 540,000 reviews. Reviews were capped at 100 tokens, with the vocabulary restricted to the 20,000 most frequent words.

Key Results and Rapid Convergence

By forcing the Source network to approximate the Target network’s encoding, the model learns a highly informed context vector for the decoder. The teacher provides such a strong signal that the student reaches high performance almost immediately.

Word-level Perplexity over Epochs

TransNet achieved a low word-level perplexity of 34.095 in just one epoch. In contrast, the previous state-of-the-art model (CF-GCN) required six epochs to reach a comparable level of performance. This represents a 4x improvement in training time for the same generation quality. Qualitatively, the model successfully captures the structure and domain-specific vocabulary of beer reviews (e.g., "Pours a dark black color...", "The aroma is of coffee...") after a single training epoch.

Limitations and Future Work

While TransNets demonstrate significant improvements in training efficiency and perplexity, a few limitations remain:

  • Evaluation Metrics: The evaluation relies heavily on word-level perplexity. While standard for language models, perplexity does not always perfectly correlate with human readability or the factual accuracy of the generated sentiment.
  • Dataset Density: The model was tested on a highly dense dataset (150-core). Its performance on "cold start" scenarios, where users or items have very few reviews, requires further investigation.

Ultimately, the student-teacher framework introduced by TransNets offers a highly efficient method for training review generators, showing strong potential for deployment on massive, real-world datasets where training time is a critical bottleneck.

Summary updated

Abstract

In recommender systems, review generation is increasingly becoming an important task. Previously proposed neural models concatenate the user and item information to each timestep of an RNN to steer it towards generating their specific review. In this paper, we show how a student-teacher like architecture can be used to rapidly build a review generator with a low perplexity score.
BibTeX
@inproceedings{catherine2018transnets,
  author = {Rose Catherine and William W. Cohen},
  title = {{TransNets for Review Generation}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
  url = {https://openreview.net/forum?id=QT4b5H-NC_D},
}
Figure