Abstract
While the role of humans is increasingly recognized in the machine learning community, representation of and interaction with models in current human-in-the-loop machine learning (HITL-ML) approaches are often too low-level and far-removed from human conceptual models. We demonstrate HEIDL, a prototype HITL-ML system that exposes the machine-learned model through high-level, explainable linguistic expressions formed of predicates representing the semantic structure of text. In HEIDL, the human's role is elevated from simply evaluating model predictions to interpreting and even updating the model logic directly by enabling interaction with rule predicates themselves. Raising the currency of interaction to such semantic levels calls for new interaction paradigms between humans and machines that result in improved productivity for text analytics model development process. Moreover, by involving humans in the process, the human-machine co-created models generalize better to unseen data as domain experts are able to instill their expertise by extrapolating from what has been learned by automated algorithms from few labeled data.
Introduction
In text analytics, a persistent trade-off exists between black-box deep learning models and manual rule-based systems. Deep learning models, such as LSTMs or Transformers, are powerful and robust to noise but lack interpretability and often fail to generalize when testing data differs from training data. Conversely, rule-based systems are transparent and reliable but require weeks of expert labor to develop manually.
Current human-in-the-loop (HITL) approaches typically treat humans as "labeling machines," providing low-level feedback on individual data points. HEIDL (Human-in-the-loop linguistic Expressions wIth Deep Learning) addresses this by raising the "currency" of interaction to high-level linguistic rules. This allows experts to inspect, validate, and modify the model's internal logic directly, ensuring it aligns with domain knowledge rather than overfitting to specific training data.
Methodology
HEIDL integrates deep learning's discovery capabilities with human domain expertise through a structured workflow that transforms raw data into production-ready rules.
The process begins with Semantic Linguistic Structure (SLS) extraction. Instead of raw tokens, the system identifies semantic roles (e.g., agent, action, theme) using a semantic role labeler. Next, the system employs differentiable logic frameworks (such as TensorLog or $\partial$ILP) to learn a set of weighted first-order logic rules from noisy data. For example, a rule might identify a communication clause if it contains a specific action in the future tense.
The HEIDL Interface
The core of HEIDL is its "playground" interface, which enables domain experts to interact with the learned rules directly.
Within this interface, users can:
- Rank and Filter: Sort rules by metrics like precision, recall, or F1 score to identify the most promising candidates.
- Modify Logic: Interactively add or remove predicates from a rule. The system provides immediate feedback on how these changes affect performance (the "delta").
- Instill Expertise: Experts can discard mathematically sound but logically nonsensical rules (overfitting) while retaining those that represent universal domain truths.
Evaluation and Results
The system was evaluated on the task of identifying "Communication" clauses in legal contracts. The training set consisted of over 28,000 sentences from IBM procurement contracts, while the test set included 1,259 sentences from non-IBM contracts to test generalization across different corporate jargon.
Key Findings
- Superior Generalization: On out-of-domain data, the HEIDL-refined model achieved a 55% F1 score, significantly outperforming both a black-box LSTM (44%) and raw machine-learned rules (41%).
- Productivity Gains: Data scientists built high-performing models in just 30 minutes, a task that typically takes weeks of manual rule writing.
- Parsimony: While the initial model generated 188 rules, experts distilled these into 5–8 high-quality rules. These smaller models were more explainable and performed better on new data.
Conclusion
HEIDL demonstrates that elevating human interaction to the level of semantic rules significantly improves both model performance and development speed. By allowing experts to identify strong inductive biases—such as recognizing that the word "notify" is a universal indicator of communication—the system creates models that generalize far better than automated approaches alone. Future work will explore extending this paradigm to more complex tasks like multi-class labeling and sentiment analysis.
Summary updated