Exploiting Rich Textual User-Product Context for Improving Personalized Sentiment Analysis

Chenyang Lyu, Linyi Yang, Yue Zhang, Yvette Graham, Jennifer FosterACL 2023 · OpenReview

Abstract

User and product information associated with a review is useful for sentiment polarity prediction. Typical approaches incorporating such information focus on modeling users and products as implicitly learned representation vectors. Most do not exploit the potential of historical reviews, or those that currently do require unnecessary modifications to model architecture or do not make full use of user/product associations. The contribution of this work is twofold: i) a method to explicitly employ historical reviews belonging to the same user/product in initializing representations, and ii) efficient incorporation of textual associations between users and products via a user-product cross-context module. Experiments on the IMDb, Yelp-2013 and Yelp-2014 English benchmarks with BERT, SpanBERT and Longformer pretrained language models show that our approach substantially outperforms previous state-of-the-art.

Introduction

Standard sentiment analysis often treats reviews as isolated pieces of text, missing the subjective nuance of personal preference. For instance, a "too spicy" rating might reflect a user's low heat tolerance rather than an objective flaw in the dish. Personalized Sentiment Analysis (PSA) aims to address this by incorporating user and product context. In their ACL 2023 paper, Lyu et al. argue that the key to better sentiment prediction lies in the "rich textual context" of historical reviews.

The Challenge of Personalization

Existing PSA models typically represent users and products as "embeddings"—vectors of numbers learned during training. However, these models often ignore the rich linguistic data in historical reviews, relying instead on abstract IDs. Furthermore, they fail to effectively model the complex relationship between users and products, such as "What kind of products does this specific user usually like?" While some previous attempts used historical text, they were often computationally expensive, requiring the maintenance of massive review matrices during training.

Methodology

The authors propose a two-pronged strategy that is both effective and computationally efficient, designed to work with Pre-trained Language Models (PLMs).

Overview of the proposed model architecture showing the integration of historical reviews and the cross-context module.

1. Text-Based Initialization

Instead of starting with random numbers, the model uses a PLM like BERT to encode all historical reviews associated with a user or product. By averaging token representations and document vectors, they create a starting "profile." A "Frobenius norm-based scaling heuristic" is applied to ensure these pre-computed vectors remain stable and compatible with the rest of the neural network.

2. User-Product Cross-Context Module

This module uses Multi-Head Attention to model four specific dimensions of interaction:

  • User-to-User: Identifying users with similar writing styles or tastes.
  • Product-to-Product: Identifying products that receive similar types of feedback.
  • User-to-Product & Product-to-User: Modeling specific preferences, such as a user's affinity for "hipster" cafes or specific flavor profiles.

Key Results

The proposed model consistently outperformed vanilla PLMs and previous state-of-the-art (SOTA) personalized models across IMDb, Yelp-2013, and Yelp-2014 benchmarks.

Accuracy comparison on the Yelp-2013 dataset across different proportions of reviews.

On the IMDb dataset, the approach improved BERT-base accuracy from 55.4% to 59.7%. The model proved robust even when users had few historical reviews by leveraging the "Product-to-Product" and "User-to-User" dimensions to "borrow" context from similar entities. Furthermore, the benefits were model-agnostic, showing consistent gains across BERT, SpanBERT, and Longformer architectures. By pre-computing historical representations, the authors also avoid the memory explosion issues that plagued earlier text-based personalization attempts.

Conclusion

This research demonstrates that historical reviews are not just records but vital context for understanding present sentiment. By explicitly "reading" past reviews and modeling cross-contextual associations, the proposed framework sets a new standard for personalized sentiment analysis while maintaining the computational efficiency required for real-world applications.

Summary updated

Abstract

User and product information associated with a review is useful for sentiment polarity prediction. Typical approaches incorporating such information focus on modeling users and products as implicitly learned representation vectors. Most do not exploit the potential of historical reviews, or those that currently do require unnecessary modifications to model architectureor do not make full use of user/product associations. The contribution of this work is twofold: i) a method to explicitly employ historical reviews belonging to the same user/product in initializing representations, and ii) efficient incorporation of textual associations between users and products via a user-product cross-context module. Experiments on the IMDb, Yelp-2013 and Yelp-2014 English benchmarks with BERT, SpanBERT and Longformer pretrained language models show that our approach substantially outperforms previous state-of-the-art.
BibTeX
@inproceedings{lyu2023exploiting,
  author = {Chenyang Lyu and Linyi Yang and Yue Zhang and Yvette Graham and Jennifer Foster},
  title = {{Exploiting Rich Textual User-Product Context for Improving Personalized Sentiment Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=4H9QIvFby9},
}

Loading related content...

Figure