SentiQ: A Probabilistic Logic Approach to Enhance Sentiment Analysis Tool Quality

Wissam Maamar Kouadri, Salima Benbernou, Mourad Ouziri, Themis Palpanas, Iheb Ben AmorarXiv 2020 · arXiv

Abstract

Sentiment analysis tools are critical for understanding consumer opinion, yet they frequently exhibit inconsistencies—both within a single tool's predictions for paraphrased text and across different tools analyzing the same document. This paper introduces SentiQ, an unsupervised framework that employs Markov Logic Networks (MLNs) to resolve these contradictions. By integrating semantic constraints and probabilistic reasoning, SentiQ identifies the most reliable sentiment predictions across a dataset. Evaluations on the Stanford Sentiment Treebank and news headline datasets show that SentiQ significantly outperforms traditional majority-voting baselines.

Introduction

Despite their widespread adoption, sentiment analysis tools are often unreliable due to two primary types of quality issues:

  1. In-tool Inconsistency: A tool may assign different polarities (positive, negative, or neutral) to semantically identical sentences (e.g., "The movie was great" vs. "The film was excellent").
  2. Inter-tool Inconsistency: Ensemble methods often face disagreements where different tools (e.g., Vader vs. Stanford Sentiment Treebank) provide conflicting scores for the same text.

While "majority voting" is a common fallback, it fails to account for the semantic relationships between documents or the varying reliability of specific tools in different contexts. SentiQ addresses these gaps by treating sentiment analysis as a probabilistic logic problem.

The SentiQ Framework

The core of SentiQ is the Markov Logic Network (MLN), which serves as a bridge between First-Order Logic and probability. The framework operates through a multi-stage pipeline designed to synthesize "prior knowledge" from multiple tools into a coherent, weighted prediction.

The SentiQ Solution Framework illustrates the multi-stage process from input clustering to MLN-based weight learning and final output generation.

The process involves several key stages:

  • Fact Generation: Unlabeled data is clustered into semantically equivalent groups. Sentiment predictions from various tools are collected as initial facts.
  • Knowledge Saturation: Logical rules infer missing information. For instance, if two documents are semantically equivalent, they should share the same polarity.
  • Inconsistency Detection: Hard constraints flag logical contradictions, such as a single document being labeled both positive and negative.
  • MLN Reasoning: The system learns weights for different rules based on their violation frequency across the entire dataset. This "global vision" allows SentiQ to assess the reliability of each tool.
  • Weighted Majority Voting: Final probabilities are generated, favoring the most consistent and reliable tool predictions.

Experimental Evaluation

SentiQ was evaluated using the News Headlines Dataset (SemEval-2017) and the Stanford Sentiment Treebank (SST). To rigorously test in-tool consistency, the researchers used a Generative Adversarial Network (GAN) to produce paraphrases, creating clusters of semantically identical sentences.

Key Results

SentiQ consistently demonstrated superior performance over standard majority-voting (MV) baselines. On a sample of news headlines, SentiQ achieved an accuracy of 0.76, a substantial improvement over the 0.50 accuracy of the inter-tool majority vote.

Comparative accuracy of SentiQ against individual sentiment analysis methods on the Stanford dataset, showing SentiQ's superior performance.

The results confirm that resolving in-tool inconsistencies—ensuring paraphrases receive matching scores—directly correlates with higher overall system accuracy. Furthermore, SentiQ proved robust even when ensemble members were weak or biased toward "neutral" predictions, as the MLN could identify and down-weight unreliable tools through logical contradictions.

Discussion and Future Work

SentiQ’s primary innovation lies in treating semantics as a logical constraint in an unsupervised setting. By looking at tool behavior globally rather than locally, it effectively filters noise from inconsistent predictions.

However, the computational cost of MLN solvers remains a challenge for scaling to massive real-time datasets. Future research will focus on optimizing inference complexity and expanding the rule set to include domain-specific expert knowledge, further refining the quality of sentiment analysis in specialized fields.

Summary updated

Abstract

The opinion expressed in various Web sites and social-media is an essential contributor to the decision making process of several organizations. Existing sentiment analysis tools aim to extract the polarity (i.e., positive, negative, neutral) from these opinionated contents. Despite the advance of the research in the field, sentiment analysis tools give \textit{inconsistent} polarities, which is harmful to business decisions. In this paper, we propose SentiQ, an unsupervised Markov logic Network-based approach that injects the semantic dimension in the tools through rules. It allows to detect and solve inconsistencies and then improves the overall accuracy of the tools. Preliminary experimental results demonstrate the usefulness of SentiQ.
BibTeX
@misc{kouadri2020sentiq,
  author = {Wissam Maamar Kouadri and Salima Benbernou and Mourad Ouziri and Themis Palpanas and Iheb Ben Amor},
  title = {{SentiQ: A Probabilistic Logic Approach to Enhance Sentiment Analysis Tool Quality}},
  year = {2020},
  eprint = {2008.08919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.08919},
}
Figure