Revisiting the Importance of Encoding Logic Rules in Sentiment Classification

Kalpesh Krishna, Preethi Jyothi, Mohit Iyyer

Abstract

We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in Hu et al. (2016), which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.

Introduction: The Challenge of Complex Sentiment

Sentiment classification often fails on syntactically complex sentences. A classic example is the "A-but-B" structure (e.g., "The acting was good, but the plot was boring"). In these cases, the sentiment of the whole sentence usually follows the sentiment of phrase "B," regardless of the sentiment expressed in phrase "A."

Previous state-of-the-art research suggested that neural networks should be explicitly taught these logic rules using techniques like knowledge distillation. However, this work investigates whether those gains were statistically significant or merely artifacts of high variance on small datasets like the Stanford Sentiment Treebank (SST2). We also explore whether modern contextualized word representations, such as ELMo, can learn these rules implicitly.

Rethinking Reproducibility in NLP

A primary finding of this study is the "reproducibility crisis" inherent in small-data benchmarks. By re-implementing logic-rule models and running them across 100 different random seeds, we observed a massive variation in accuracy—up to 4 percentage points—just by changing the seed.

Accuracy distribution and training curves over 100 random seeds, highlighting the significant variance in model performance.

As shown in Figure 1, reporting a single "best" number on datasets like SST2 is misleading. When averaged over many runs, the previously reported benefits of "distillation" (teaching a model via a rule-based teacher) largely vanish. Most of the actual gains attributed to logic rules in prior work came from "projection"—a simpler post-processing step that forces the model to obey the rule—rather than the distillation process itself.

Logic Rules vs. Contextualized Embeddings

While explicit logic rules provide marginal gains, switching to ELMo-based models significantly improves performance on A-but-B sentences by approximately 8% over a baseline CNN. Crucially, ELMo achieves this without ever being explicitly told the logic rule.

Visualization of ELMo's internal states reveals that it naturally learns to group words within the "A" and "B" segments differently. The model effectively "discovers" the syntax of contrastive conjunctions through its pre-training on large-scale corpora.

Heatmaps of ELMo's internal states showing how the model implicitly segments and processes contrastive 'A-but-B' structures.

Statistical testing (Kolmogorov-Smirnov) confirms that ELMo's predictions are significantly closer to the "logic rule manifold" than standard models. This suggests that contextualized embeddings are not just better feature extractors, but are capable of capturing complex linguistic logic implicitly.

The Bottleneck of Human Ambiguity

To further understand model failures, we conducted a crowdsourced analysis where humans rated the sentiment and ambiguity of A-but-B sentences. The results indicate that many instances where models fail are also instances where humans disagree. As models improve, they are increasingly hitting a ceiling defined by inherent linguistic ambiguity rather than simple logical errors.

Conclusion

This study serves as a reminder to the NLP community regarding the necessity of rigorous statistical reporting. We demonstrate that contextualized embeddings like ELMo implicitly capture complex linguistic rules far more effectively than hard-coded logic distillation. Future work should continue to explore what other logical properties are "baked into" newer architectures like BERT and its successors, while maintaining high standards for reproducibility.

Summary updated

Abstract

We analyze the performance of different sentiment classification models on syntactically complex inputs like A-but-B sentences. The first contribution of this analysis addresses reproducible research: to meaningfully compare different models, their accuracies must be averaged over far more random seeds than what has traditionally been reported. With proper averaging in place, we notice that the distillation model described in arXiv:1603.06318v4 [cs.LG], which incorporates explicit logic rules for sentiment classification, is ineffective. In contrast, using contextualized ELMo embeddings (arXiv:1802.05365v2 [cs.CL]) instead of logic rules yields significantly better performance. Additionally, we provide analysis and visualizations that demonstrate ELMo's ability to implicitly learn logic rules. Finally, a crowdsourced analysis reveals how ELMo outperforms baseline models even on sentences with ambiguous sentiment labels.
BibTeX
@inproceedings{krishna2018revisiting,
  author = {Kalpesh Krishna and Preethi Jyothi and Mohit Iyyer},
  title = {{Revisiting the Importance of Encoding Logic Rules in Sentiment Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=HJZvYffOZS},
}

Loading related content...

Figure