HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop

Prithviraj Sen, Yunyao Li, Eser Kandogan, Yiwei Yang, Walter S. Lasecki

Introduction

Modern text analytics faces a persistent trade-off between the high performance of "black box" deep learning models and the interpretability of traditional rule-based systems. While deep learning excels at capturing complex patterns, it often suffers from poor out-of-domain generalization. Conversely, rule-based systems are transparent but brittle, requiring significant manual effort to maintain logical consistency across noisy datasets.

HEIDL (Human-in-the-loop Linguistic Expression Learning) addresses this gap by elevating the human role from a simple labeler to a co-creator of model logic. Instead of just providing more data, domain experts use HEIDL to refine machine-generated linguistic rules, ensuring they are both accurate and generalizable.

The HEIDL workflow: transitioning from deep learning to human-refined production rules.

Methodology

The HEIDL system integrates automated rule induction with a sophisticated interactive interface through three primary components:

1. Semantic Linguistic Structure (SLS)

The system extracts shallow semantic representations from text, identifying "who did what to whom." These representations form the basis of predicates—logical building blocks that include verb tense, voice, and dictionary-based matches for roles such as agent or theme.

2. Deep Learning for Rule Induction

HEIDL utilizes differentiable logic (e.g., TensorLog or $\partial$ILP) to learn weighted logical rules from noisy, labeled data. This allows the machine to propose an initial set of high-quality candidate rules that are inherently interpretable.

3. Interactive UI and the "Playground"

The core of HEIDL is its interactive interface, which allows users to manage the rule-learning process. Users can rank and filter hundreds of machine-generated rules based on standard metrics like precision, recall, and F1 score. The "Playground" feature enables users to manually add or remove predicates from a rule, providing immediate feedback on how these changes affect performance across the dataset.

The HEIDL interactive interface, allowing users to rank, filter, and modify linguistic rules in the 'Playground'.

Experimental Evaluation

The system was evaluated on a real-world legal contract classification task, specifically identifying sentences related to "Communication" (e.g., notice transmission protocols).

  • Training Set: 28,174 sentences from 149 IBM procurement contracts.
  • Test Set (Out-of-Domain): 1,259 sentences from 11 non-IBM procurement contracts.

This out-of-domain test is critical for assessing generalization, as legal formatting and terminology vary significantly between organizations.

Results and Discussion

HEIDL demonstrated significant improvements in both model performance and developer productivity:

  • Superior Generalization: While a standard black-box Bi-LSTM model achieved a 44% F1 score on out-of-domain data, human-machine co-created models via HEIDL reached 52% to 55% F1, representing a 25% relative improvement.
  • Productivity Gains: Data scientists built high-performing models in just 30 minutes. Manually authoring these rules by reviewing the 28,000+ training sentences would typically take weeks.
  • Parsimony and Robustness: Human-led teams achieved better results using only 5–8 clean, logical rules compared to the 188 weighted rules used by the deep learning baseline. This suggests that domain experts are better at identifying universal logic than machines, which often overfit to dataset-specific quirks.

Conclusion

HEIDL successfully bridges the gap between deep learning and rule-based systems by putting the human "in the loop" of the logic-creation process. While the current system relies on predefined predicates and requires domain expertise, it proves that collaborative human-AI development can produce models that are more robust, interpretable, and efficient to build than either humans or machines could create alone. Future work will explore extending this framework to more complex information extraction tasks and expanding the predicate discovery process.

Summary updated

Abstract

While the role of humans is increasingly recognized in machine learning community, representation of and interaction with models in current human-in-the-loop machine learning (HITL-ML) approaches are too low-level and far-removed from human's conceptual models. We demonstrate HEIDL, a prototype HITL-ML system that exposes the machine-learned model through high-level, explainable linguistic expressions formed of predicates representing semantic structure of text. In HEIDL, human's role is elevated from simply evaluating model predictions to interpreting and even updating the model logic directly by enabling interaction with rule predicates themselves. Raising the currency of interaction to such semantic levels calls for new interaction paradigms between humans and machines that result in improved productivity for text analytics model development process. Moreover, by involving humans in the process, the human-machine co-created models generalize better to unseen data as domain experts are able to instill their expertise by extrapolating from what has been learned by automated algorithms from few labelled data.
BibTeX
@inproceedings{sen2019heidl,
  author = {Prithviraj Sen and Yunyao Li and Eser Kandogan and Yiwei Yang and Walter S. Lasecki},
  title = {{HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Eq7E7hvQnB},
}
Figure