Abstract
Sentiment analysis tools are critical for understanding consumer opinion, yet they frequently exhibit inconsistencies—both within a single tool's predictions for paraphrased text and across different tools analyzing the same document. This paper introduces SentiQ, an unsupervised framework that employs Markov Logic Networks (MLNs) to resolve these contradictions. By integrating semantic constraints and probabilistic reasoning, SentiQ identifies the most reliable sentiment predictions across a dataset. Evaluations on the Stanford Sentiment Treebank and news headline datasets show that SentiQ significantly outperforms traditional majority-voting baselines.
Introduction
Despite their widespread adoption, sentiment analysis tools are often unreliable due to two primary types of quality issues:
- In-tool Inconsistency: A tool may assign different polarities (positive, negative, or neutral) to semantically identical sentences (e.g., "The movie was great" vs. "The film was excellent").
- Inter-tool Inconsistency: Ensemble methods often face disagreements where different tools (e.g., Vader vs. Stanford Sentiment Treebank) provide conflicting scores for the same text.
While "majority voting" is a common fallback, it fails to account for the semantic relationships between documents or the varying reliability of specific tools in different contexts. SentiQ addresses these gaps by treating sentiment analysis as a probabilistic logic problem.
The SentiQ Framework
The core of SentiQ is the Markov Logic Network (MLN), which serves as a bridge between First-Order Logic and probability. The framework operates through a multi-stage pipeline designed to synthesize "prior knowledge" from multiple tools into a coherent, weighted prediction.
The process involves several key stages:
- Fact Generation: Unlabeled data is clustered into semantically equivalent groups. Sentiment predictions from various tools are collected as initial facts.
- Knowledge Saturation: Logical rules infer missing information. For instance, if two documents are semantically equivalent, they should share the same polarity.
- Inconsistency Detection: Hard constraints flag logical contradictions, such as a single document being labeled both positive and negative.
- MLN Reasoning: The system learns weights for different rules based on their violation frequency across the entire dataset. This "global vision" allows SentiQ to assess the reliability of each tool.
- Weighted Majority Voting: Final probabilities are generated, favoring the most consistent and reliable tool predictions.
Experimental Evaluation
SentiQ was evaluated using the News Headlines Dataset (SemEval-2017) and the Stanford Sentiment Treebank (SST). To rigorously test in-tool consistency, the researchers used a Generative Adversarial Network (GAN) to produce paraphrases, creating clusters of semantically identical sentences.
Key Results
SentiQ consistently demonstrated superior performance over standard majority-voting (MV) baselines. On a sample of news headlines, SentiQ achieved an accuracy of 0.76, a substantial improvement over the 0.50 accuracy of the inter-tool majority vote.
The results confirm that resolving in-tool inconsistencies—ensuring paraphrases receive matching scores—directly correlates with higher overall system accuracy. Furthermore, SentiQ proved robust even when ensemble members were weak or biased toward "neutral" predictions, as the MLN could identify and down-weight unreliable tools through logical contradictions.
Discussion and Future Work
SentiQ’s primary innovation lies in treating semantics as a logical constraint in an unsupervised setting. By looking at tool behavior globally rather than locally, it effectively filters noise from inconsistent predictions.
However, the computational cost of MLN solvers remains a challenge for scaling to massive real-time datasets. Future research will focus on optimizing inference complexity and expanding the rule set to include domain-specific expert knowledge, further refining the quality of sentiment analysis in specialized fields.
Summary updated