Disentangling Aspect and Stance via a Siamese Autoencoder for Aspect Clustering of Vaccination Opinions

Lixing Zhu, Runcong Zhao, Gabriele Pergola, Yulan He

Introduction

Monitoring public sentiment regarding vaccinations is a critical task for public health officials, particularly as new concerns and misinformation emerge in real-time on social media. However, traditional text-mining approaches often struggle to categorize these opinions effectively. This paper introduces Disentangled Opinion Clustering (DOC), a novel framework designed to separate the specific topic of a post (the aspect) from the author's sentiment toward that topic (the stance). By isolating these two components, the DOC model achieves superior performance in clustering vaccination-related discourse, even when encountering previously unseen topics.

The Challenge: Stance Interference in Opinion Mining

Public opinion on vaccines is highly dynamic. New issues—such as mRNA technology, fertility concerns, or booster mandates—frequently surface. Supervised models trained on fixed labels are often unable to detect these emerging topics.

Furthermore, standard text-clustering methods suffer from "stance interference." For instance, consider two tweets: "mRNA vaccines are poison" and "The Pfizer vaccine is safe." Both discuss the aspect of safety, yet their stances are diametrically opposed. Traditional models often place these into different clusters because the linguistic markers of stance (e.g., "poison" vs. "safe") outweigh the markers of the shared topic. This interference creates fragmented clusters, making it difficult for researchers to analyze the full breadth of the debate surrounding a single issue.

Methodology: Disentangled Opinion Clustering (DOC)

The DOC model addresses stance interference through a two-stage training process centered on a Siamese Autoencoder architecture.

1. Unsupervised Pre-training

The model begins with a DeBERTa-based encoder-decoder structure. It is pre-trained on a massive unlabelled corpus of 4.37 million vaccination-related tweets. Using a masked token prediction task, the model learns the general linguistic patterns and vocabulary specific to the vaccination domain.

2. Supervised Disentanglement and the Swapping Mechanism

The core innovation of DOC is the separation of a sentence's representation into two distinct latent vectors: $z_s$ (representing stance) and $z_a$ (representing aspect). To ensure these vectors are truly independent, the researchers implemented a Swapping Autoencoder strategy.

The DOC training pipeline: (a) Unsupervised pre-training, (b) Supervised disentanglement of stance ($z_s$) and aspect ($z_a$), and (c) the Swapped AutoEncoder mechanism.

As illustrated in the figure above, the model takes pairs of tweets that share the same aspect but differ in stance. It then swaps the aspect embeddings ($z_a$) between the two tweets. The decoder is then challenged to reconstruct the original text using these swapped representations. If the reconstruction is successful, it demonstrates that the aspect vector $z_a$ captures the topic information independently of the stance. Additionally, the model utilizes Sentence-BERT (SBERT) as an inductive bias to help the aspect vector focus on the specific text spans that define the topic.

Data and Experimental Setup

The model was evaluated using two primary public Twitter datasets:

  • CMF (Covid-Moral-Foundation): Tweets categorized by moral foundations (e.g., Care/Harm, Liberty/Oppression).
  • VAD (Vaccination Attitude Detection): Tweets covering eight specific aspect categories, such as Side Effects, Immunity Level, and Economic Effects.

Results and Performance

The DOC model demonstrated significant improvements over state-of-the-art baselines, including SBERT and specialized clustering models like SCCL.

  • Clustering Accuracy: On the VAD dataset, DOC achieved a Clustering Accuracy (CA) of 66.7%, outperforming SBERT-k-means (60.5%).
  • Zero-Shot Generalization: One of the model's greatest strengths is its ability to generalize. When trained on the CMF dataset and tested on the VAD dataset (which contains different topics), DOC maintained a high performance of 56.2% CA. This suggests the model learns the nature of how aspects are expressed rather than just memorizing specific keywords.
  • Latent Space Visualization: t-SNE projections revealed that DOC’s aspect vectors ($z_a$) form tight, distinct clusters. In contrast, standard embeddings often result in a "hairball" where different topics are inextricably mixed due to stance interference.

Conclusion and Future Work

The DOC model provides a robust solution for disentangling stance from aspect in social media analysis. By utilizing a swapping autoencoder, the framework allows for cleaner, more actionable topic clustering that is resilient to the "noise" of polarized sentiments.

While the model currently requires a pre-defined number of clusters ($k$) and a large pre-training corpus, it represents a significant step forward in automated opinion mining. Future research will explore methods for automatically determining optimal cluster counts and applying this disentanglement framework to other highly polarized domains, such as political discourse and product reviews.

Summary updated

Abstract

Mining public opinions about vaccines from social media has been increasingly relevant to analyse trends in public debates and to provide quick insights to policy-makers. However, the application of existing models has been hindered by the wide variety of users’ attitudes and the new aspects continuously arising in the public debate. Existing approaches, frequently framed via well-known tasks, such as aspect classification or text span detection, make direct usage of the supervision information constraining the models to predefined aspect classes, while still not distinguishing those aspects from users’ stances. As a result, this has significantly hindered the dynamic integration of new aspects. We thus propose a model, namely Disentangled Opinion Clustering (DOC), for vaccination opinion mining from social media. DOC is able to disentangle users’ stances from opinions via a disentangling attention mechanism and a Swapping-Autoencoder, and is designed to process unseen aspect categories via a clustering approach, leveraging clustering-friendly representations induced by out-of-the-box Sentence-BERT encodings and disentangling mechanisms. We conduct a thorough experimental assessment demonstrating the benefit of the disentangling mechanisms and cluster-based approach on both the quality of aspect clusters and the generalization across new aspect categories, outperforming existing methodologies on aspect-based opinion mining.
BibTeX
@inproceedings{zhu2023disentangling,
  author = {Lixing Zhu and Runcong Zhao and Gabriele Pergola and Yulan He},
  title = {{Disentangling Aspect and Stance via a Siamese Autoencoder for Aspect Clustering of Vaccination Opinions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.115},
}
Figure