Introduction
Modern text analytics faces a persistent trade-off between the high performance of "black box" deep learning models and the interpretability of traditional rule-based systems. While deep learning excels at capturing complex patterns, it often suffers from poor out-of-domain generalization. Conversely, rule-based systems are transparent but brittle, requiring significant manual effort to maintain logical consistency across noisy datasets.
HEIDL (Human-in-the-loop Linguistic Expression Learning) addresses this gap by elevating the human role from a simple labeler to a co-creator of model logic. Instead of just providing more data, domain experts use HEIDL to refine machine-generated linguistic rules, ensuring they are both accurate and generalizable.
Methodology
The HEIDL system integrates automated rule induction with a sophisticated interactive interface through three primary components:
1. Semantic Linguistic Structure (SLS)
The system extracts shallow semantic representations from text, identifying "who did what to whom." These representations form the basis of predicates—logical building blocks that include verb tense, voice, and dictionary-based matches for roles such as agent or theme.
2. Deep Learning for Rule Induction
HEIDL utilizes differentiable logic (e.g., TensorLog or $\partial$ILP) to learn weighted logical rules from noisy, labeled data. This allows the machine to propose an initial set of high-quality candidate rules that are inherently interpretable.
3. Interactive UI and the "Playground"
The core of HEIDL is its interactive interface, which allows users to manage the rule-learning process. Users can rank and filter hundreds of machine-generated rules based on standard metrics like precision, recall, and F1 score. The "Playground" feature enables users to manually add or remove predicates from a rule, providing immediate feedback on how these changes affect performance across the dataset.
Experimental Evaluation
The system was evaluated on a real-world legal contract classification task, specifically identifying sentences related to "Communication" (e.g., notice transmission protocols).
- Training Set: 28,174 sentences from 149 IBM procurement contracts.
- Test Set (Out-of-Domain): 1,259 sentences from 11 non-IBM procurement contracts.
This out-of-domain test is critical for assessing generalization, as legal formatting and terminology vary significantly between organizations.
Results and Discussion
HEIDL demonstrated significant improvements in both model performance and developer productivity:
- Superior Generalization: While a standard black-box Bi-LSTM model achieved a 44% F1 score on out-of-domain data, human-machine co-created models via HEIDL reached 52% to 55% F1, representing a 25% relative improvement.
- Productivity Gains: Data scientists built high-performing models in just 30 minutes. Manually authoring these rules by reviewing the 28,000+ training sentences would typically take weeks.
- Parsimony and Robustness: Human-led teams achieved better results using only 5–8 clean, logical rules compared to the 188 weighted rules used by the deep learning baseline. This suggests that domain experts are better at identifying universal logic than machines, which often overfit to dataset-specific quirks.
Conclusion
HEIDL successfully bridges the gap between deep learning and rule-based systems by putting the human "in the loop" of the logic-creation process. While the current system relies on predefined predicates and requires domain expertise, it proves that collaborative human-AI development can produce models that are more robust, interpretable, and efficient to build than either humans or machines could create alone. Future work will explore extending this framework to more complex information extraction tasks and expanding the predicate discovery process.
Summary updated