Extractive Opinion Summarization in Quantized Transformer Spaces

Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, Mirella Lapata

Introduction

Opinion summarization presents a unique challenge in Natural Language Processing. Unlike news summarization, which typically follows a single factual narrative, opinion summarization must aggregate a vast and often conflicting distribution of subjective attitudes from hundreds or thousands of users.

Current approaches face two major hurdles:

  1. Lack of Supervision: Obtaining human-written "gold standard" summaries for thousands of reviews is prohibitively expensive, necessitating unsupervised models.
  2. The Hallucination Trap: Abstractive models, which generate new text, often hallucinate facts or incorrectly use first-person pronouns, undermining the summary's reliability.

The Quantized Transformer (QT) addresses these issues by providing an extractive, unsupervised, and scalable framework for summarizing large-scale opinion data.

Methodology: The Quantized Transformer

The core of this approach is the Quantized Transformer (QT), which adapts the architecture of a Vector-Quantized Variational Autoencoder (VQ-VAE) for text processing.

Architecture

The QT utilizes a Transformer encoder to map input sentences into a multi-head representation. Each "head" is quantized by mapping it to the nearest "code" (a centroid) within a learned codebook. During training, a decoder attempts to reconstruct the original sentence from these discrete codes, forcing the model to learn a meaningful latent space.

The Quantized Transformer architecture maps sentences to discrete latent codes for reconstruction.

Popularity-Driven Extraction

During inference, the decoder is bypassed. Instead, the model leverages the quantized space to identify salient opinions. If multiple sentences from different reviews map to the same latent code, that code is considered representative of a "popular" opinion.

The extraction process follows a two-step sampling strategy:

  1. Cluster Sampling: A "popular" code is selected based on the frequency of sentences assigned to it.
  2. Sentence Sampling: A specific sentence is selected that is geometrically closest to the chosen code in the vector space.

This ensures that the resulting summary is both representative of the general consensus and linguistically high-quality.

The extraction pipeline: sentences are encoded, clustered by popularity in the latent space, and sampled to form a summary.

Controllable Aspect Summarization

A significant finding of this work is that the multi-head structure of the Transformer naturally disentangles different attributes (e.g., sentiment vs. topic). By using a small set of seed words (e.g., "clean," "staff"), the model can identify specific heads or codes corresponding to particular aspects. This allows for "zero-shot" aspect-specific summarization without requiring additional training.

The SPACE Dataset

To evaluate the model, the authors introduced SPACE, a large-scale benchmark for opinion summarization consisting of 1.1 million TripAdvisor hotel reviews. Unlike previous datasets like Yelp or Amazon, SPACE provides 1,050 human-written summaries for 50 hotels, including both general and aspect-specific summaries across six categories: building, cleanliness, food, location, rooms, and service.

Key Results and Insights

  • Superior Performance: QT significantly outperformed competitive baselines, including extractive models like LexRank and abstractive models like MeanSum, in ROUGE-L scores on the SPACE dataset.
  • Human Preference: In "Best-Worst Scaling" evaluations, human judges preferred QT summaries for their informativeness and lack of redundancy.
  • Aspect Accuracy: QT's aspect-specific summaries were rated as highly exclusive to their topics, achieving 58.7% exclusivity compared to just 6.7% for the leading abstractive baseline.
  • Data Efficiency: The model remains highly effective even when trained on only 5% of the available data, demonstrating significant practical advantages over data-hungry neural models.

Conclusion and Future Work

While QT excels at faithfulness and salience, the extractive nature of the model can sometimes lead to "jumpy" transitions between sentences, resulting in lower coherence scores compared to abstractive models. Future research could explore integrating the popularity-driven logic of quantized spaces into abstractive frameworks to combine the fluency of generation with the factual grounding of extraction.

Reference: Angelidis, S., Amplayo, R. K., Suhara, Y., Wang, X., & Lapata, M. (2021). Extractive Opinion Summarization in Quantized Transformer Spaces. Transactions of the Association for Computational Linguistics, 9, 277–293.

Summary updated

Abstract

We present the Quantized Transformer (QT), an unsupervised system for extractive opinion summarization. QT is inspired by Vector-Quantized Variational Autoencoders, which we repurpose for popularity-driven summarization. It uses a clustering interpretation of the quantized space and a novel extraction algorithm to discover popular opinions among hundreds of reviews, a significant step towards opinion summarization of practical scope. In addition, QT enables controllable summarization without further training, by utilizing properties of the quantized space to extract aspect-specific summaries. We also make publicly available SPACE, a large-scale evaluation benchmark for opinion summarizers, comprising general and aspect-specific summaries for 50 hotels. Experiments demonstrate the promise of our approach, which is validated by human studies where judges showed clear preference for our method over competitive baselines.
BibTeX
@article{angelidis2021extractive,
  author = {Stefanos Angelidis and Reinald Kim Amplayo and Yoshihiko Suhara and Xiaolan Wang and Mirella Lapata},
  title = {{Extractive Opinion Summarization in Quantized Transformer Spaces}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00366},
}
Figure