Towards Quantifying the Distance between Opinions

Saket Gurukar, Deepak Ajwani, Sourav Dutta, Juho Lauri, Srinivasan Parthasarathy, Alessandra Sala

Abstract

Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a large number of opinions on a topic, there is a necessity for automated tools to help navigate the space of opinions. In such contexts understanding and quantifying the similarity between opinions is key. We find that measures based solely on text similarity or on overall sentiment often fail to effectively capture the distance between opinions. Thus, we propose a new distance measure for capturing the similarity between opinions that leverages the nuanced observation -- similar opinions express similar sentiment polarity on specific relevant entities-of-interest. Specifically, in an unsupervised setting, our distance measure achieves significantly better Adjusted Rand Index scores (up to 56x) and Silhouette coefficients (up to 21x) compared to existing approaches. Similarly, in a supervised setting, our opinion distance measure achieves considerably better accuracy (up to 20% increase) compared to extant approaches that rely on text similarity, stance similarity, and sentiment similarity.

Introduction

Measuring the "distance" between two opinions is a fundamental challenge in natural language processing. While standard tools excel at measuring lexical or grammatical similarity, they often fail to distinguish between diametrically opposed viewpoints that use similar vocabulary. For instance, consider two statements: "Hillary looked presidential while Trump came across as manipulative" and "Trump looked presidential while Hillary came across as manipulative." To a bag-of-words model or even many semantic embedding models, these sentences appear nearly identical. However, their underlying stances are opposite. This paper addresses the inadequacy of existing text-similarity and document-sentiment measures for navigating complex public opinion spaces.

Methodology

The authors propose a novel "opinion distance" measure ($OD$) that represents an opinion as a set of (subject, sentiment) pairs. The core idea is that opinion similarity is defined by the alignment of sentiment toward specific entities.

Overview of the Opinion Distance (OD) calculation pipeline.

The researchers developed two primary variants for this representation:

  1. OD-parse: Utilizes dependency parsing to identify noun phrases as subjects and links them to sentiment-bearing words (adjectives, adverbs, verbs).
  2. OD (Disambiguated Concept Representation): A more robust version that maps phrases to Wikipedia entities. This allows the system to recognize synonyms (e.g., "Video games" and "Electronic games") as the same subject. It employs a "distance-weighted" sentiment approach where words closer to the subject in a sentence exert a stronger influence on its sentiment score.

To calculate the distance between two opinions, the system creates a bipartite graph of discussed subjects and uses Word Mover Distance (WMD) to find the optimal semantic mapping. The final distance is the average difference in sentiment polarity across these mapped subjects.

Evaluation and Results

The method was evaluated across 19 datasets, including 18 stance detection datasets from the IBM Debater project and a new, nuanced "Seanad Abolition" dataset regarding the Irish Parliament. This latter dataset is particularly challenging as it contains three distinct expert-curated clusters, including a "reform but don't abolish" middle ground.

Clustering performance (ARI and Silhouette) of OD compared to standard NLP baselines.

The proposed $OD$ measure significantly outperformed baselines in both unsupervised and supervised tasks. In unsupervised clustering, $OD$ achieved an Adjusted Rand Index (ARI) of 0.56 on the "Video Games" dataset, whereas traditional models like TF-IDF, WMD, and BERT performed near random levels (ARI $\approx$ 0). In supervised classification tasks, using $OD$ as a feature for an SVM classifier resulted in accuracy gains of up to 20% over traditional text-similarity features.

Discussion and Conclusion

The key insight of this work is that sentiment direction is paramount: opinion similarity is not about the words used, but the alignment of sentiment toward specific entities. The failure of state-of-the-art embeddings like BERT to capture these nuances highlights the need for specialized opinion distance measures. While the $OD$ method faces challenges such as computational complexity and potential disambiguation errors, it provides a powerful tool for researchers and policymakers to navigate the contradictory landscape of digital public opinion by focusing on how things are said about specific subjects.

Summary updated

Abstract

Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a large number of opinions on a topic, there is a necessity for automated tools to help navigate the space of opinions. In such contexts understanding and quantifying the similarity between opinions is key. We find that measures based solely on text similarity or on overall sentiment often fail to effectively capture the distance between opinions. Thus, we propose a new distance measure for capturing the similarity between opinions that leverages the nuanced observation -- similar opinions express similar sentiment polarity on specific relevant entities-of-interest. Specifically, in an unsupervised setting, our distance measure achieves significantly better Adjusted Rand Index scores (up to 56x) and Silhouette coefficients (up to 21x) compared to existing approaches. Similarly, in a supervised setting, our opinion distance measure achieves considerably better accuracy (up to 20% increase) compared to extant approaches that rely on text similarity, stance similarity, and sentiment similarity
BibTeX
@inproceedings{gurukar2020towards,
  author = {Saket Gurukar and Deepak Ajwani and Sourav Dutta and Juho Lauri and Srinivasan Parthasarathy and Alessandra Sala},
  title = {{Towards Quantifying the Distance between Opinions}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=IlGhlveGrB},
}
Figure