Abstract
We present the Quantized Transformer (QT), an unsupervised system for extractive opinion summarization. QT is inspired by Vector-Quantized Variational Autoencoders, which we repurpose for popularity-driven summarization. It uses a clustering interpretation of the quantized space and a novel extraction algorithm to discover popular opinions among hundreds of reviews, a significant step towards opinion summarization of practical scope. In addition, QT enables controllable summarization without further training, by utilizing properties of the quantized space to extract aspect-specific summaries. We also make publicly available SPACE, a large-scale evaluation benchmark for opinion summarizers, comprising general and aspect-specific summaries for 50 hotels. Experiments demonstrate the promise of our approach, which is validated by human studies where judges showed clear preference for our method over competitive baselines.
Introduction
Opinion summarization faces a unique challenge in natural language processing: the lack of large-scale "gold standard" training data. Unlike news summarization, where headlines or lead paragraphs often serve as summaries, user reviews rarely come with human-written synopses. Consequently, models must operate in an unsupervised manner.
Existing state-of-the-art models often favor abstractive approaches, which generate new text. However, these models frequently suffer from "hallucinations" (generating factually incorrect information) or awkward phrasing that fails to represent the collective voice of hundreds of reviewers. Furthermore, most current systems are limited to processing a small number of reviews (typically around 10), whereas real-world entities like popular hotels or products often have thousands. The Quantized Transformer (QT) addresses these issues by providing an extractive framework that scales to hundreds of reviews and allows for fine-grained aspect control.
The Quantized Transformer (QT)
Architecture and Training
The core of QT is a Transformer-based architecture that incorporates a discretization bottleneck, similar to Vector-Quantized Variational Autoencoders (VQ-VAE).
During training, a Transformer encoder maps sentences into multi-head representations. These continuous vectors are then passed through a quantization layer that maps them to a fixed set of "latent codes" (centroids). A decoder then attempts to reconstruct the original sentence from these discrete codes. This process forces the model to learn a compressed, clustered representation of the input space, where similar opinions are grouped around the same latent codes.
Popularity-Driven Extraction
At inference time, the decoder is discarded, and the model focuses on the learned latent space to perform extraction. To generate a summary, QT identifies the most representative sentences through a two-step sampling process:
- Cluster Sampling: The model assigns all reviews for a specific entity to their nearest latent codes. It then selects codes based on their "popularity"—the number of sentences assigned to them.
- Sentence Sampling: For each selected code, the model picks a specific sentence from the original reviews that is geometrically closest to that code in the vector space.
This approach avoids the common pitfall of averaging review vectors, which often "blurs" distinct opinions into a generic, uninformative representation. By treating the latent space as a collection of distinct opinion groups, QT preserves the sharpness and specificity of popular praises or complaints.
Aspect-Specific Control
One of the most significant findings of this work is that the multi-head nature of Transformers can be exploited for controllability without retraining. Different "heads" in the Transformer naturally capture different types of information (e.g., sentiment vs. topic). By using a minimal set of keywords (as few as 5 per aspect), the system can identify which "sub-space" corresponds to a specific aspect like "cleanliness" or "service." Sampling can then be restricted to these clusters to produce targeted summaries.
Experimental Results and the SPACE Corpus
To evaluate the model, the authors introduced the SPACE corpus, a massive new benchmark containing 1.1 million TripAdvisor reviews for training and 1,050 human-written summaries for 50 hotels. This dataset covers both general overviews and six specific aspects (Building, Cleanliness, Food, Location, Rooms, Service).
Quantitative Performance
On the SPACE dataset, QT achieved a ROUGE-L score of 21.90, significantly outperforming strong abstractive baselines such as Copycat (20.90) and MeanSum (19.92). The model also proved remarkably data-efficient, maintaining high performance even when trained on only 5% of the available data.
Human Evaluation
In "Best-Worst Scaling" tests, human judges overwhelmingly preferred QT summaries over competitors. The model showed a clear lead in informativeness (+50.7) and non-redundancy (+18.0). Furthermore, in a study of aspect specificity, 58.7% of QT's aspect-specific summaries were judged to discuss the requested aspect exclusively, compared to just 6.7% for the leading abstractive competitor.
Conclusion
The Quantized Transformer represents a significant step forward in making opinion summarization practical for real-world applications. By leveraging vector quantization for clustering and exploiting the multi-head structure of Transformers for aspect control, QT provides a robust, extractive alternative to generative models. While transitions between extracted sentences can occasionally feel less cohesive than generated text, the gains in factual reliability and scalability make it a powerful tool for navigating large volumes of consumer feedback.
Summary updated