Capturing User and Product Information for Document Level Sentiment Analysis with Deep Memory Network

Zi-Yi DouEMNLP 2017 · OpenReview

Abstract

Sentiment analysis often overlooks the context of the reviewer and the product being reviewed. This paper introduces the User Product Deep Memory Network (UPDMN), an architecture that leverages an attention-based memory mechanism to incorporate historical user preferences and product quality. By performing multiple computational "hops" over a memory bank of past reviews, UPDMN significantly outperforms traditional text-only models and previous state-of-the-art methods on large-scale datasets like IMDB and Yelp.

Introduction

Standard sentiment analysis treats documents as independent entities. However, sentiment is inherently subjective: a "lenient" user might rate a mediocre product highly, while a "fastidious" critic might give the same product a low score. Similarly, products have inherent quality baselines. To accurately predict a star rating, a model must understand these biases.

Previous models, such as the User Product Neural Network (UPNN), attempted to address this using fixed vectors for users and products. However, these methods struggle with data sparsity when users have few reviews. UPDMN addresses this by using a memory network to store and attend to actual historical document representations, providing a more robust way to "remember" context even when data is sparse.

Methodology: The User Product Deep Memory Network

The UPDMN architecture consists of two main components: document embedding and a deep memory network.

Document Embedding

The model first processes the review text using a Long Short-Term Memory (LSTM) network. It generates a document representation ($q$) by averaging the hidden states of all words in the review.

Deep Memory Network and Multiple Hops

The core innovation is the memory mechanism. The model maintains a memory bank containing representations of other reviews written by the same user ($U(d)$) and reviews for the same product ($P(d)$).

The UPDMN architecture illustrating the multi-hop memory mechanism.

As shown in the architecture diagram, the model employs a "content-based attention" layer to compare the current document vector against the memory. It performs multiple "hops"—computational layers where the document representation is iteratively refined by combining it with weighted information retrieved from memory. This allows the model to progressively filter noise and focus on the most relevant historical context before the final softmax prediction.

Experimental Results

The model was evaluated on the IMDB, Yelp 2013, and Yelp 2014 datasets. These datasets provide the necessary metadata to link reviews to specific users and products.

Performance comparison of UPDMN against various baselines across three datasets.

As detailed in the results table, UPDMN consistently outperforms a wide range of baselines, including SVMs, Paragraph Vectors, and the previous state-of-the-art UPNN. On the IMDB dataset, UPDMN(4) reached an accuracy of 0.465, compared to 0.435 for UPNN and 0.399 for standard trigram models.

The results also demonstrate the "hop" advantage: increasing the number of hops generally improves performance up to a point (typically 4 hops), as the model learns to better filter noise from user and product histories. Furthermore, the model achieved lower Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), indicating that its predictions are closer to the ground truth even when the exact star rating is missed.

Conclusion and Future Work

The UPDMN effectively captures user and product context to "normalize" sentiment, handling data sparsity better than fixed-vector approaches. While the multi-hop attention mechanism provides a clear advantage, performance can degrade with excessive hops (e.g., 5 hops on some datasets) due to overfitting. Future research could explore more sophisticated hierarchical text encoders or complex attention strategies to further enhance the model's ability to extract predictive signals from historical data.

Summary updated

BibTeX
@inproceedings{dou2017capturing,
  author = {Zi-Yi Dou},
  title = {{Capturing User and Product Information for Document Level Sentiment Analysis with Deep Memory Network}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=H1ZGBfGdbH},
}
Figure