TL;DR
Standard text similarity and sentiment analysis fail to capture the true distance between human opinions, which often share identical vocabularies yet argue completely opposing viewpoints. This paper introduces an unsupervised Opinion Distance ($OD$) metric that grounds opinions in canonical entities, evaluates the sentiment polarity directed at each entity, and aligns them across texts. In both clustering and classification benchmarks, this subject-sentiment formulation dramatically outperforms baselines like BERT, Doc2vec, and TF-IDF.
What Problem Does This Paper Address?
Automated moderation, policy analysis, and digital democracy require clustering and navigating thousands of public comments. However, existing NLP metrics struggle with opinions for two primary reasons:
- High lexical overlap: Opposing statements often mirror each other structurally (e.g., "Hillary looked presidential while Trump came across as manipulative" vs. "Trump looked presidential while Hillary came across as manipulative"). Traditional text similarity metrics (TF-IDF, Doc2vec, BERT) treat them as nearly identical.
- Stance is too coarse: Stance detection simply flags text as for or against a topic. Two people might support Brexit for radically contradictory reasons (e.g., economic nationalism vs. curbing immigration), yet coarse stance models group them together.
The authors set out to define a principled, unsupervised distance measure specifically tailored to the semantics of opinions.
Methods
The core hypothesis is simple: similar opinions express similar sentiment polarities toward the same or semantically related subjects. The proposed pipeline computes distance in three stages:
[Opinion Text] -> [Disambiguate Subjects (TagMe)] -> [Score Contextual Polarity]
|
[Align Subjects via WMD]
|
[Aggregate Polarity Differences] -> Distance [0, 1]
- Opinion Subject Extraction & Disambiguation: Rather than relying entirely on brittle dependency parse trees (an alternative evaluated as
OD-parse), the primary model (OD) uses TagMe to link noun phrases to canonical Wikipedia concepts. This handles polysemy and synonyms gracefully (e.g., mapping "PC game" and "electronic game" to the same conceptual space). - Contextual Polarity Scoring: Surrounding sentiment words (derived from the IBM Debater lexicon) are weighted using an inverse token distance penalty ($1/\sqrt{d}$) relative to the subject. A curated list of polarity shifters (e.g., neither, rarely, reduce) flips signs appropriately. The sentiment toward a subject is compressed into a normalized scalar.
- Subject Mapping & Polarity Aggregation: Subjects between two opinions are mapped into a bipartite graph. Earth/Word Mover's Distance (WMD) finds a minimum-weight matching based on concept embeddings (Word2vec or Doc2vec), pruning matches exceeding a semantic distance threshold.
- Distance Computation: The distance is calculated as the average absolute difference between the polarities of matched subject pairs, scaled between $[0, 1]$.
Datasets
- IBM Debater Benchmarks: 18 stance-detection datasets covering controversial debate topics (e.g., Video Games, Gambling, Atheism, Monarchy, Affirmative Action). The underlying assumption is that shared stance generally aligns with shared opinions.
- Seanad Abolition (Civiq.eu): A newly curated, nuanced 3-cluster opinion dataset (25 claims) debating whether Ireland should abolish its upper house of parliament, reform it, or keep it temporarily until the lower house (Dáil) is reformed.
- BBC News Comments: Real-world reader comments on Brexit political articles used for qualitative case studies.
Key Insights and Ideas
- Syntactic parsing hurts more than it helps: The parse-tree approach (
OD-parse) performed poorly because syntax parsers frequently stumble on messy social comments and miss implicit sentiment targets. Reweighting proximity words around disambiguated Wikipedia entities (OD) proved vastly superior. - Polarity shifters are vital: Discarding negators and polarity shifters collapsed clustering performance back toward random chance on several datasets (e.g., dropping ARI on Video Games from 0.56 to 0.08).
- Continuous polarity distributions underperform scalars: Modeling sentiment as a full distribution and computing distances via Jensen-Shannon Divergence or Earth Mover's Distance yielded worse clusters than using aggregated, discretized polarity values.
Key Results
- Unsupervised Clustering: In Adjusted Rand Index (ARI) and Silhouette evaluations across 18 datasets,
ODdrastically outperformed traditional baselines: - On Video Games,
ODachieved an ARI of 0.56 compared to 0.08 for BERT, 0.01 for WMD, and -0.01 for TF-IDF. - On Pornography,
ODreached an ARI of 0.41 compared to -0.01 for BERT and -0.02 for TF-IDF. - Across all 18 topics,
ODaveraged an ARI of 0.20–0.22, while standard text baselines hovered around 0.01–0.09. - Supervised Multi-Class Classification: Using distance matrices as feature vectors in an SVM classifier,
ODalone reached an F1-score of 0.88 on Video Games and Pornography. CombiningODwith n-grams or LSA achieved top performance across benchmarks (up to a 20% gain over lexical baselines).
Limitations and Future Work
- Reliance on Entity Linking: If entity linking fails, the method degrades. On Hydroelectric Dams,
ODunderperformed TF-IDF because TagMe struggled to disambiguate domain-specific references like "hydro." - Threshold Sensitivity & Undefined Distances: If two documents share no subjects within the semantic distance threshold, their opinion distance remains undefined.
- Data Scarcity: The field severely lacks rich, multi-faceted ground truth opinion benchmarks; most existing sets are binary stance datasets rather than spectrums of human thought. The authors plan to build dedicated, large-scale nuanced opinion corpora in future work.
Summary updated