Abstract
Opinion summarization aims to condense multiple documents into a single representative text. However, social media data—such as tweets and Reddit threads—presents significant challenges due to its noisy, ungrammatical, and stylistically diverse nature. We present WassOS, an unsupervised model that treats documents as probability distributions within the Wasserstein space. By explicitly disentangling semantic meaning from syntactic style and utilizing the Wasserstein barycenter to identify the mathematical "center of mass" of opinions, WassOS produces summaries that preserve core meaning more effectively than previous state-of-the-art methods. Evaluations across Twitter, Reddit, and Amazon datasets demonstrate superior performance in both automated ROUGE metrics and human-rated meaning preservation.
Introduction
The proliferation of user-generated content on social media has created a need for robust opinion summarization tools. While existing models perform well on structured reviews (e.g., Amazon), they often falter when applied to the "noise" of platforms like Twitter or Reddit. In these domains, posts are frequently ungrammatical and laden with slang.
Traditional models typically represent documents in Euclidean space and use simple averaging to generate a summary representation. This approach often conflates the "what" (semantics) with the "how" (syntax), leading to summaries that are a "confusing soup" of conflicting writing styles rather than a clear synthesis of opinions. Furthermore, standard distance metrics like KL-divergence can suffer from vanishing gradients when comparing non-overlapping distributions. WassOS addresses these issues by leveraging the geometric properties of the Wasserstein space.
Methodology: The WassOS Framework
WassOS introduces three primary technical innovations to handle noisy text clusters:
1. Disentangled Latent Space
Using a Variational Autoencoder (VAE), the model splits a document's internal representation into two distinct latent spaces: one for semantics and one for syntax. This separation ensures that the core opinion is not diluted by the idiosyncratic writing styles of individual users.
2. Wasserstein Barycenter
Instead of calculating a simple vector average, WassOS treats each document as a distribution. The summary is computed as the Wasserstein barycenter—the optimal point that minimizes the Wasserstein distance to all individual document distributions in the cluster. This metric is more robust to noise and provides a smoother geometry for optimization compared to Euclidean or KL-divergence-based methods.
3. BERT-Guided Decoder
To translate the mathematical summary back into natural language, the model employs a GRU decoder enhanced with a Transformer layer. This layer is initialized with parameters from the middle layers of BERT, which are specifically chosen for their ability to capture strong syntactic features, thereby improving the grammaticality of the generated text.
Experimental Setup and Datasets
The model was evaluated on three diverse datasets to test its generalizability:
- Twitter: 2,214 clusters of tweets concerning COVID-19 and political events.
- Reddit: A new dataset of 40 threads from
r/COVID19_support, featuring expert-written gold-standard summaries. - Amazon: A standard benchmark consisting of 60 product review clusters.
Results and Discussion
WassOS demonstrated significant improvements over previous state-of-the-art models like Copycat and COOP. On the Twitter dataset, the model showed a relative improvement of 5% to 11% in ROUGE scores. On Reddit, it achieved the highest ROUGE-1 and ROUGE-2 scores among compared models.
In human evaluations, WassOS was particularly noted for its Meaning Preservation. Across all three domains, human judges found that WassOS was significantly better at capturing the actual gist of the source documents. Ablation studies further confirmed that both the disentanglement of latent spaces and the Transformer-based decoder were critical to the model's success; removing either component resulted in a marked decline in summary quality.
Conclusion
WassOS represents a step forward in unsupervised summarization by moving beyond simple Euclidean averaging. By treating opinions as distributions in Wasserstein space, the model successfully navigates the noise of social media text. While the model currently faces some challenges with sentence redundancy—likely due to the beam search algorithm and latent feeding mechanisms—it provides a robust framework for capturing global properties in diverse document clusters. Future work will focus on scaling the model to larger clusters and addressing scenarios involving diametrically opposed viewpoints within a single cluster.
Summary updated