Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification

Prithviraj Sen, Marina Danilevsky, Yunyao Li, Siddhartha Brahma, Matthias Boehm, Laura Chiticariu, Rajasekar KrishnamurthyEMNLP 2020 · OpenReview

In the landscape of modern Natural Language Processing (NLP), a persistent tension exists between model performance and interpretability. While deep learning models like LSTMs and Transformers achieve state-of-the-art accuracy, they often operate as "black boxes," making it difficult for users to understand the reasoning behind specific classifications. This lack of transparency is particularly problematic in high-stakes domains such as legal contract analysis or medical diagnostics.

At EMNLP 2020, researchers introduced RuleNN, a neuro-symbolic framework designed to bridge this gap. RuleNN learns transparent first-order logic (FOL) rules for sentence classification, matching the performance of complex recurrent networks while providing human-readable and editable linguistic expressions (LEs).

The RuleNN Architecture

RuleNN is a modular neural network architecture that treats explainability as a core requirement rather than an afterthought. The system follows a structured pipeline that transforms raw text into verifiable logical rules.

The RuleNN pipeline: from raw sentences and dictionaries to verified linguistic expressions.

The process begins with a Natural Language Understanding (NLU) component that extracts facts and predicates from input sentences using semantic role labeling (SRL). These predicates represent linguistic features, such as the specific verb used or the semantic role of a noun phrase.

The core of the system is the RuleNN module, which consists of two primary sub-modules:

  1. Predicate Generation Module (PGM): This module learns weighted combinations of linguistic features.
  2. Clause Generation Module (CGM): These modules combine predicates into logical conjunctions (AND statements). To maintain differentiability for training via gradient descent, RuleNN employs "smooth t-norms" to approximate logical operations.

Multiple CGMs are aggregated using a MAX operation, representing logical disjunction (OR). After training, the continuous weights are converted back into "crisp" FOL rules for human inspection and verification.

Bridging Language and Logic

A key strength of RuleNN is its ability to translate complex linguistic structures into clear logical expressions. By leveraging SRL, the model can identify the underlying "who did what to whom" in a sentence, which serves as the foundation for its rules.

An example of a sentence parsed into semantic roles and its corresponding first-order logic rule.

As shown in the example above, a sentence like "Notices may be transmitted electronically..." is decomposed into its semantic components. RuleNN then learns a rule—such as identifying a "communication" if the sentence contains a specific verb (inform or transmit) and the object (A1) is a "notice." This level of granularity allows domain experts to see exactly why a sentence was classified into a particular category.

Experimental Evaluation

The researchers evaluated RuleNN against two distinct datasets: the standard TREC question classification benchmark and a challenging Contracts dataset consisting of legal sentences. The Contracts dataset is particularly demanding due to extreme class imbalance and the requirement for out-of-domain generalization.

Performance and Explainability

RuleNN demonstrated significant advantages over traditional symbolic methods. On the Contracts dataset, its Area Under the Precision-Recall Curve (AUC-PR) was 6.8x to 7.6x higher than traditional Inductive Logic Programming (ILP) and statistical relational learning baselines.

More importantly, RuleNN remained competitive with black-box models. It performed comparably to a Bi-directional LSTM (BiLSTM) and even outperformed it on 4 out of 9 labels in the legal dataset. Unlike the BiLSTM, however, RuleNN's decisions are fully traceable.

Human-in-the-Loop Refinement

One of the most compelling aspects of RuleNN is its support for human-machine co-creation. In a user study, data scientists used a graphical interface to review and refine the 188 rules learned by RuleNN. By filtering and tweaking these rules, the experts created a "compressed" model that actually outperformed the original neural network. The human-refined models achieved an F-measure approximately 25% higher than the BiLSTM, highlighting the power of combining human intuition with transparent AI.

Conclusion

RuleNN represents a significant step toward accountable NLP. By constraining neural networks to a logical structure, it achieves high performance with far fewer parameters than traditional deep learning models. While it currently faces some bottlenecks related to the accuracy of initial NLU parsing, it provides a robust blueprint for building AI systems that are both powerful and understandable. For specialized domains where "why" is as important as "what," RuleNN offers a path forward for collaborative intelligence.

Summary updated

BibTeX
@inproceedings{sen2020learning,
  author = {Prithviraj Sen and Marina Danilevsky and Yunyao Li and Siddhartha Brahma and Matthias Boehm and Laura Chiticariu and Rajasekar Krishnamurthy},
  title = {{Learning Explainable Linguistic Expressions with Neural Inductive Logic Programming for Sentence Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=YrnVNy8vowU},
}
Figure