Casting the Same Sentiment Classification Problem

Erik Körner, Ahmad Dawar Hakimi, Gerhard Heyer, Martin PotthastEMNLP 2021 · DOI

Introduction

Traditional sentiment analysis typically focuses on absolute polarity—classifying a single text as positive, negative, or neutral. However, this approach often struggles with domain shifts; a model trained on restaurant reviews might learn that "delicious" is a strong positive signal but fail when applied to electronics reviews where "reliable" is the key descriptor.

This paper introduces the "same sentiment classification problem": a task where a model must determine if two texts share the same sentiment polarity without necessarily identifying what that sentiment is. By reframing sentiment analysis as a pairwise similarity task, the authors hypothesize that "sameness" is a more universal trait that relies less on topic-specific vocabulary, making models more robust when moving between different subjects.

Methodology

The researchers adapted the standard sequence pair classification approach using the BERT (Bidirectional Encoder Representations from Transformers) architecture. Specifically, they utilized BERT-base-uncased with a custom classification head consisting of a dropout layer followed by a dense layer with a sigmoid activation.

Instead of a single sentence, the model receives two reviews separated by a special token. The model then uses the representation of the [CLS] token to predict a binary outcome: "same sentiment" or "different sentiment." This approach was compared against traditional machine learning baselines (SVM and Logistic Regression using TF-IDF/Doc2Vec) and Siamese Bi-LSTM networks using GloVe embeddings.

Experimental Setup

The study primarily utilized the Yelp Business Review Dataset, which contains roughly 6.7 million reviews across 22 main business categories. Star ratings (1–5) were converted into binary labels: "good" (4–5 stars) and "bad" (1–3 stars).

The core of the data preparation was the pairing strategy: random pairs of reviews were matched about the same business. This ensures the model cannot rely on the topic to find a match; it must look specifically at the sentiment expressed about that topic. The authors found that increasing the sequence length to 512 tokens was necessary to capture enough nuance in longer business reviews.

Key Insights and Results

The best BERT-based model achieved an overall accuracy of 89.05% across all categories. One of the most significant findings was the "Good-Good" bias: the model was significantly better at identifying two positive reviews as "same" (91.2% accuracy) than identifying two negative reviews as "same" (84.6%). This suggests that linguistic markers of "goodness" are more consistent than the varied ways people express "badness."

Table 1: Performance metrics for different review pairings, highlighting the higher accuracy for 'good-good' pairs.

Accuracy varied by topic, with "Financial Services" hitting a high of 95.05% and "Arts & Entertainment" at a lower 83.82%. Despite these variations, the model maintained high accuracy even when training and testing sets were from completely different categories, supporting the idea of topic agnosticism in sentiment agreement.

Table 2: General statistics and evaluation metrics across 22 Yelp business categories.

Conclusion and Future Work

Reframing sentiment as a similarity task allows for strong cross-domain generalization. While BERT outperformed Siamese Bi-LSTM and traditional ML models, the approach is computationally intensive. Future work should explore lighter transformer variants like DistilBERT or ALBERT to reduce overhead. Additionally, investigating whether this "sameness" approach works for fine-grained sentiment (e.g., distinguishing between "okay" and "amazing") remains an open area for research.

Summary updated

BibTeX
@inproceedings{korner2021casting,
  author = {Erik Körner and Ahmad Dawar Hakimi and Gerhard Heyer and Martin Potthast},
  title = {{Casting the Same Sentiment Classification Problem}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.53},
}
Figure