Revisiting the Importance of Encoding Logic Rules in Sentiment Classification

Kalpesh Krishna, Preethi Jyothi, Mohit Iyyer

Abstract

We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in arXiv:1603.06318v4 [cs.LG], which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings (arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.

Introduction: The Challenge of Complex Sentiment

Sentiment analysis often struggles with syntactically complex sentences, such as "A-but-B" structures (e.g., "The plot was predictable, but the acting was phenomenal"). While a simple bag-of-words model might be confused by conflicting sentiment words, humans recognize that the overall sentiment typically follows the second phrase ("B").

Previous research, notably Hu et al. (2016), proposed "harnessing" these logic rules by forcing neural networks to follow them through techniques like knowledge distillation and posterior projection. This paper investigates whether such explicit rules are necessary or if modern contextualized embeddings, like ELMo, can implicitly learn these structures more effectively than manual programming.

Methodology: The Reproducibility Framework

A core contribution of this work is the introduction of a rigorous reproducibility framework. Deep learning models often exhibit significant performance variance based on their random initialization (the "random seed"). To ensure findings were not mere statistical flukes, the authors ran every experiment 100 times and averaged the results.

Accuracy variance across 100 random seeds.

As shown in Figure 1, the accuracy of a single model run can fluctuate by nearly 4% depending on the seed. This "variance trap" suggests that many reported state-of-the-art results in NLP may be over-reported due to a lack of proper statistical averaging. The authors used the Kolmogorov-Smirnov test to confirm that while ELMo and projection provide significant gains, the distillation method provided no statistically significant improvement when properly averaged.

Key Insights: Implicit vs. Explicit Rules

The study evaluates three main approaches using a Convolutional Neural Network (CNN) baseline: logic rule projection, iterative distillation, and contextualized embeddings (ELMo).

The results indicate that ELMo does not just perform better overall; it specifically excels at handling "A-but-B" sentences without being explicitly taught the rule. By generating word representations based on the entire sentence context, ELMo implicitly learns that the words following "but" carry more weight for the final sentiment. Visual analysis using cosine similarity heatmaps confirms that ELMo "groups" the decisive parts of a sentence together in vector space, providing visual evidence of this implicit learning.

Key Results: Debunking the Gains

When subjected to rigorous averaging, the previously reported benefits of explicit logic rules largely vanished.

Comparison of reported vs. averaged test accuracy.

The original Hu et al. (2016) paper reported a +1.6% accuracy gain from logic rules. However, as shown in Figure 2, averaging over 100 seeds reduced this gain to a mere +0.29%. In contrast, simply switching to ELMo embeddings improved the baseline by 2.9% overall. On specific "A-but-B" sentences, ELMo boosted accuracy from ~78.7% to ~86.5%, far outperforming explicit logic rule methods (~83.4%).

Furthermore, a crowdsourced analysis revealed a striking correlation: the sentences that models get wrong are often the same ones that human crowd-workers find ambiguous or neutral, suggesting that errors in automated systems are often tied to inherent linguistic ambiguity.

Conclusion

This research highlights the critical importance of statistical averaging in NLP research to avoid the variance trap. It demonstrates that modern contextualized embeddings are capable of implicitly learning complex linguistic rules that were previously thought to require explicit programming. Future work should continue to explore how these vectors represent logical complexity across various natural language tasks beyond sentiment analysis.

Summary updated

Abstract

We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in arXiv:1603.06318v4 [cs.LG], which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings (arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.
BibTeX
@inproceedings{krishna2018revisiting,
  author = {Kalpesh Krishna and Preethi Jyothi and Mohit Iyyer},
  title = {{Revisiting the Importance of Encoding Logic Rules in Sentiment Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=HJZvYffOZS},
}
Figure