TransSum: Translating Aspect and Sentiment Embeddings for Self-Supervised Opinion Summarization

Ke Wang, Xiaojun Wan

Abstract

Opinion summarization aims to generate concise summaries from collections of user reviews without the need for human-annotated labels. This paper introduces TransSum, a self-supervised framework that treats opinion summarization as a "translation" between disentangled aspect and sentiment embeddings. By leveraging contrastive learning to exploit product-level aspect invariance and sentiment-level invariance across different products, TransSum creates high-quality synthetic training data. Experimental results on Yelp, Amazon, and Rotten Tomatoes datasets demonstrate that TransSum significantly outperforms existing unsupervised and self-supervised baselines in both automated metrics and human evaluations.

Introduction

Summarizing user reviews (opinion summarization) is a significant challenge in NLP because, unlike news articles, gold-standard human-written summaries are rarely available for training. Most existing methods are unsupervised, often relying on autoencoders to compress reviews into a latent space. However, these methods face two major hurdles: redundancy, where models struggle to merge repeating points efficiently, and relevance, where synthetic training data often pairs unrelated reviews, leading to hallucinations.

Methodology

TransSum addresses these challenges by breaking a review down into two distinct components: Aspect (the topic) and Sentiment (the opinion). The framework assumes a review representation ($c$) is the sum of its aspect ($a$) and sentiment ($s$): $c = a + s$.

Disentangled Embeddings via Contrastive Learning

The model uses a Transformer encoder to project reviews into two separate low-dimensional spaces. To ensure these spaces accurately represent their respective components, the authors employ two novel contrastive objectives:

  • Aspect Invariance: Reviews of the same product (intra-group) should have similar aspect embeddings, even if the sentiment differs.
  • Sentiment Invariance: Reviews with the same sentiment label (e.g., "Positive"), even for different products (inter-group), should have similar sentiment embeddings.
Conceptual overview of aspect and sentiment spaces and their respective invariances.

Translation-Based Review Modeling and Summarization

The architecture consists of a translation-based modeling component and a multi-input summarization component. The encoder $E$ produces disentangled vectors $a_i$ and $s_i$, which are combined to reconstruct the original review. For summarization, the model aggregates embeddings from multiple reviews to generate a coherent summary via a decoder $D$.

The TransSum architecture, showing translation-based review modeling (left) and multi-input opinion summarization (right).

High-Relevance Synthetic Dataset

To train the final summarizer, TransSum avoids random review pairing. Instead, it uses the learned aspect embeddings to find a subset of reviews that discuss the same topics as a "target" review. This target review then serves as a high-quality pseudo-summary for that specific subset, creating a more realistic and grounded supervised training signal.

Experiments and Results

The model was evaluated on three diverse domains: Yelp (business reviews), Amazon (product reviews), and Rotten Tomatoes (movie reviews). While training is entirely self-supervised, evaluation is performed against human-written summaries.

Key Findings

  • State-of-the-Art Performance: TransSum outperformed several strong baselines, including MeanSum, CopyCat, and PlanSum. On the Yelp dataset, it achieved a ROUGE-L of 20.31, notably surpassing the extractive "Upper Bound" of 18.11.
  • Human Evaluation: In "Best-Worst Scaling" tests, human judges found TransSum summaries to be significantly more relevant and less redundant than those from competing abstractive models.
  • Ablation Success: Removing either the aspect or sentiment contrastive loss led to a measurable drop in performance, proving that both types of invariance are necessary for high-quality summarization.

Discussion and Future Work

The success of TransSum highlights that disentanglement is key: by separating "what" from "how," the model can perform weighted pooling in the embedding space to cancel out redundant information. Furthermore, looking across products to learn general sentiment patterns strengthens the model's robustness.

Future work could explore more sophisticated dataset distillation techniques to further improve synthetic data quality. Additionally, moving beyond binary sentiment labels to more granular scales (e.g., 1-5 stars) and exploring controllable summarization (e.g., focusing on specific aspects like price) remain promising directions.

Summary updated

BibTeX
@inproceedings{wang2021transsum,
  author = {Ke Wang and Xiaojun Wan},
  title = {{TransSum: Translating Aspect and Sentiment Embeddings for Self-Supervised Opinion Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.65},
}
Figure