HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop

Yiwei Yang, Eser Kandogan, Yunyao Li, Walter S. Lasecki, Prithviraj Sen

Abstract

While the role of humans is increasingly recognized in the machine learning community, representation of and interaction with models in current human-in-the-loop machine learning (HITL-ML) approaches are often too low-level and far-removed from human conceptual models. We demonstrate HEIDL, a prototype HITL-ML system that exposes the machine-learned model through high-level, explainable linguistic expressions formed of predicates representing the semantic structure of text. In HEIDL, the human's role is elevated from simply evaluating model predictions to interpreting and even updating the model logic directly by enabling interaction with rule predicates themselves. Raising the currency of interaction to such semantic levels calls for new interaction paradigms between humans and machines that result in improved productivity for text analytics model development process. Moreover, by involving humans in the process, the human-machine co-created models generalize better to unseen data as domain experts are able to instill their expertise by extrapolating from what has been learned by automated algorithms from few labeled data.

Introduction

In text analytics, a persistent trade-off exists between black-box deep learning models and manual rule-based systems. Deep learning models, such as LSTMs or Transformers, are powerful and robust to noise but lack interpretability and often fail to generalize when testing data differs from training data. Conversely, rule-based systems are transparent and reliable but require weeks of expert labor to develop manually.

Current human-in-the-loop (HITL) approaches typically treat humans as "labeling machines," providing low-level feedback on individual data points. HEIDL (Human-in-the-loop linguistic Expressions wIth Deep Learning) addresses this by raising the "currency" of interaction to high-level linguistic rules. This allows experts to inspect, validate, and modify the model's internal logic directly, ensuring it aligns with domain knowledge rather than overfitting to specific training data.

Methodology

HEIDL integrates deep learning's discovery capabilities with human domain expertise through a structured workflow that transforms raw data into production-ready rules.

The HEIDL workflow: from deep learning discovery to human-refined production models.

The process begins with Semantic Linguistic Structure (SLS) extraction. Instead of raw tokens, the system identifies semantic roles (e.g., agent, action, theme) using a semantic role labeler. Next, the system employs differentiable logic frameworks (such as TensorLog or $\partial$ILP) to learn a set of weighted first-order logic rules from noisy data. For example, a rule might identify a communication clause if it contains a specific action in the future tense.

The HEIDL Interface

The core of HEIDL is its "playground" interface, which enables domain experts to interact with the learned rules directly.

The HEIDL user interface, featuring rule ranking, predicate selection, and real-time performance feedback.

Within this interface, users can:

  • Rank and Filter: Sort rules by metrics like precision, recall, or F1 score to identify the most promising candidates.
  • Modify Logic: Interactively add or remove predicates from a rule. The system provides immediate feedback on how these changes affect performance (the "delta").
  • Instill Expertise: Experts can discard mathematically sound but logically nonsensical rules (overfitting) while retaining those that represent universal domain truths.

Evaluation and Results

The system was evaluated on the task of identifying "Communication" clauses in legal contracts. The training set consisted of over 28,000 sentences from IBM procurement contracts, while the test set included 1,259 sentences from non-IBM contracts to test generalization across different corporate jargon.

Key Findings

  • Superior Generalization: On out-of-domain data, the HEIDL-refined model achieved a 55% F1 score, significantly outperforming both a black-box LSTM (44%) and raw machine-learned rules (41%).
  • Productivity Gains: Data scientists built high-performing models in just 30 minutes, a task that typically takes weeks of manual rule writing.
  • Parsimony: While the initial model generated 188 rules, experts distilled these into 5–8 high-quality rules. These smaller models were more explainable and performed better on new data.

Conclusion

HEIDL demonstrates that elevating human interaction to the level of semantic rules significantly improves both model performance and development speed. By allowing experts to identify strong inductive biases—such as recognizing that the word "notify" is a universal indicator of communication—the system creates models that generalize far better than automated approaches alone. Future work will explore extending this paradigm to more complex tasks like multi-class labeling and sentiment analysis.

Summary updated

Abstract

While the role of humans is increasingly recognized in machine learning community, representation of and interaction with models in current human-in-the-loop machine learning (HITL-ML) approaches are too low-level and far-removed from human's conceptual models. We demonstrate HEIDL, a prototype HITL-ML system that exposes the machine-learned model through high-level, explainable linguistic expressions formed of predicates representing semantic structure of text. In HEIDL, human's role is elevated from simply evaluating model predictions to interpreting and even updating the model logic directly by enabling interaction with rule predicates themselves. Raising the currency of interaction to such semantic levels calls for new interaction paradigms between humans and machines that result in improved productivity for text analytics model development process. Moreover, by involving humans in the process, the human-machine co-created models generalize better to unseen data as domain experts are able to instill their expertise by extrapolating from what has been learned by automated algorithms from few labelled data.
BibTeX
@inproceedings{yang2019heidl,
  author = {Yiwei Yang and Eser Kandogan and Yunyao Li and Walter S. Lasecki and Prithviraj Sen},
  title = {{HEIDL: Learning Linguistic Expressions with Deep Learning and Human-in-the-Loop}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Eq7E7hvQnB},
}
Figure