Towards Quantifying the Distance between Opinions

Saket Gurukar, Deepak Ajwani, Sourav Dutta, Juho Lauri, Srinivasan Parthasarathy, Alessandra Sala

TL;DR

Standard text similarity and sentiment analysis fail to capture the true distance between human opinions, which often share identical vocabularies yet argue completely opposing viewpoints. This paper introduces an unsupervised Opinion Distance ($OD$) metric that grounds opinions in canonical entities, evaluates the sentiment polarity directed at each entity, and aligns them across texts. In both clustering and classification benchmarks, this subject-sentiment formulation dramatically outperforms baselines like BERT, Doc2vec, and TF-IDF.

What Problem Does This Paper Address?

Automated moderation, policy analysis, and digital democracy require clustering and navigating thousands of public comments. However, existing NLP metrics struggle with opinions for two primary reasons:

  1. High lexical overlap: Opposing statements often mirror each other structurally (e.g., "Hillary looked presidential while Trump came across as manipulative" vs. "Trump looked presidential while Hillary came across as manipulative"). Traditional text similarity metrics (TF-IDF, Doc2vec, BERT) treat them as nearly identical.
  2. Stance is too coarse: Stance detection simply flags text as for or against a topic. Two people might support Brexit for radically contradictory reasons (e.g., economic nationalism vs. curbing immigration), yet coarse stance models group them together.

The authors set out to define a principled, unsupervised distance measure specifically tailored to the semantics of opinions.

Methods

The core hypothesis is simple: similar opinions express similar sentiment polarities toward the same or semantically related subjects. The proposed pipeline computes distance in three stages:

[Opinion Text] -> [Disambiguate Subjects (TagMe)] -> [Score Contextual Polarity] 
                                                               |
                                            [Align Subjects via WMD]
                                                               |
                                            [Aggregate Polarity Differences] -> Distance [0, 1]
  1. Opinion Subject Extraction & Disambiguation: Rather than relying entirely on brittle dependency parse trees (an alternative evaluated as OD-parse), the primary model (OD) uses TagMe to link noun phrases to canonical Wikipedia concepts. This handles polysemy and synonyms gracefully (e.g., mapping "PC game" and "electronic game" to the same conceptual space).
  2. Contextual Polarity Scoring: Surrounding sentiment words (derived from the IBM Debater lexicon) are weighted using an inverse token distance penalty ($1/\sqrt{d}$) relative to the subject. A curated list of polarity shifters (e.g., neither, rarely, reduce) flips signs appropriately. The sentiment toward a subject is compressed into a normalized scalar.
  3. Subject Mapping & Polarity Aggregation: Subjects between two opinions are mapped into a bipartite graph. Earth/Word Mover's Distance (WMD) finds a minimum-weight matching based on concept embeddings (Word2vec or Doc2vec), pruning matches exceeding a semantic distance threshold.
  4. Distance Computation: The distance is calculated as the average absolute difference between the polarities of matched subject pairs, scaled between $[0, 1]$.

Datasets

  • IBM Debater Benchmarks: 18 stance-detection datasets covering controversial debate topics (e.g., Video Games, Gambling, Atheism, Monarchy, Affirmative Action). The underlying assumption is that shared stance generally aligns with shared opinions.
  • Seanad Abolition (Civiq.eu): A newly curated, nuanced 3-cluster opinion dataset (25 claims) debating whether Ireland should abolish its upper house of parliament, reform it, or keep it temporarily until the lower house (Dáil) is reformed.
  • BBC News Comments: Real-world reader comments on Brexit political articles used for qualitative case studies.

Key Insights and Ideas

  • Syntactic parsing hurts more than it helps: The parse-tree approach (OD-parse) performed poorly because syntax parsers frequently stumble on messy social comments and miss implicit sentiment targets. Reweighting proximity words around disambiguated Wikipedia entities (OD) proved vastly superior.
  • Polarity shifters are vital: Discarding negators and polarity shifters collapsed clustering performance back toward random chance on several datasets (e.g., dropping ARI on Video Games from 0.56 to 0.08).
  • Continuous polarity distributions underperform scalars: Modeling sentiment as a full distribution and computing distances via Jensen-Shannon Divergence or Earth Mover's Distance yielded worse clusters than using aggregated, discretized polarity values.

Key Results

  • Unsupervised Clustering: In Adjusted Rand Index (ARI) and Silhouette evaluations across 18 datasets, OD drastically outperformed traditional baselines:
  • On Video Games, OD achieved an ARI of 0.56 compared to 0.08 for BERT, 0.01 for WMD, and -0.01 for TF-IDF.
  • On Pornography, OD reached an ARI of 0.41 compared to -0.01 for BERT and -0.02 for TF-IDF.
  • Across all 18 topics, OD averaged an ARI of 0.20–0.22, while standard text baselines hovered around 0.01–0.09.
  • Supervised Multi-Class Classification: Using distance matrices as feature vectors in an SVM classifier, OD alone reached an F1-score of 0.88 on Video Games and Pornography. Combining OD with n-grams or LSA achieved top performance across benchmarks (up to a 20% gain over lexical baselines).

Limitations and Future Work

  • Reliance on Entity Linking: If entity linking fails, the method degrades. On Hydroelectric Dams, OD underperformed TF-IDF because TagMe struggled to disambiguate domain-specific references like "hydro."
  • Threshold Sensitivity & Undefined Distances: If two documents share no subjects within the semantic distance threshold, their opinion distance remains undefined.
  • Data Scarcity: The field severely lacks rich, multi-faceted ground truth opinion benchmarks; most existing sets are binary stance datasets rather than spectrums of human thought. The authors plan to build dedicated, large-scale nuanced opinion corpora in future work.

Summary updated

Abstract

Increasingly, critical decisions in public policy, governance, and business strategy rely on a deeper understanding of the needs and opinions of constituent members (e.g. citizens, shareholders). While it has become easier to collect a large number of opinions on a topic, there is a necessity for automated tools to help navigate the space of opinions. In such contexts understanding and quantifying the similarity between opinions is key. We find that measures based solely on text similarity or on overall sentiment often fail to effectively capture the distance between opinions. Thus, we propose a new distance measure for capturing the similarity between opinions that leverages the nuanced observation – similar opinions express similar sentiment polarity on specific relevant entities-of-interest. Specifically, in an unsupervised setting, our distance measure achieves significantly better Adjusted Rand Index scores (up to 56x) and Silhouette coefficients (up to 21x) compared to existing approaches. Similarly, in a supervised setting, our opinion distance measure achieves considerably better accuracy (up to 20% increase) compared to extant approaches that rely on text similarity, stance similarity, and sentiment similarity.
BibTeX
@inproceedings{gurukar2020towards,
  author = {Saket Gurukar and Deepak Ajwani and Sourav Dutta and Juho Lauri and Srinivasan Parthasarathy and Alessandra Sala},
  title = {{Towards Quantifying the Distance between Opinions}},
  booktitle = {ICWSM},
  year = {2020},
  url = {https://openreview.net/forum?id=IlGhlveGrB},
}

Loading related content...

Figure