Abstract
In production environments, Natural Language Understanding (NLU) datasets are dynamic entities where annotation guidelines and language usage evolve over time. This evolution often results in conflicting labels for identical text inputs. This paper presents a novel pipeline that utilizes Information Gain to detect temporal guideline shifts and a hybrid resolution strategy—combining majority voting with a supervised LSTM model. Evaluated on a massive real-world dataset, the approach achieves a 2.75% relative error reduction.
Introduction
Standard NLU research typically treats datasets as static, assuming a fixed set of guidelines and annotators. However, in commercial voice assistant systems like Alexa, datasets are "living" entities. As new features are introduced, annotation guidelines are updated to reflect more granular or different semantic interpretations.
For instance, as shown in the figure above, the phrase "turn on" might be initially labeled as Other (O) but later re-classified as an ActionTrigger (AT). When utterances are re-sampled over time, the training set accumulates these contradictory labels. Existing conflict resolution methods often fail to scale to millions of utterances or ignore the temporal nature of guideline changes. This work proposes a scalable solution to identify when guidelines changed and determine the correct label for the current system state.
Methodology
The proposed solution is a multi-stage pipeline designed to automate data cleaning and label reconciliation.
1. Max Information Gain (IG) Cut
To address guideline evolution, the system treats the annotation timestamp as a feature. It identifies a specific date $d$ that maximizes the Information Gain between labels created before and after that point. If a significant "cut" is detected, annotations preceding that date are considered obsolete and are discarded in favor of the newer regime.
2. Normalized Entropy (NH) Filtering
Following the temporal cut, the "messiness" of the remaining labels is quantified using Normalized Entropy (NH).
- Low Entropy (NH ≤ 0.75): In cases where annotators largely agree, simple Majority Voting is employed to select the final label.
- High Entropy (NH > 0.75): When disagreement is high, majority voting becomes unreliable. These cases trigger a more sophisticated resolver.
3. LSTM-based Resolution
For high-conflict scenarios, a multi-task Bi-LSTM model (incorporating a CNN layer for character-level features) is utilized. The model is pre-trained on the unambiguous, "clean" portions of the dataset. When a conflict occurs, the model's prediction for Intent and Slot labels is adopted as the new "gold" standard.
Experimental Setup
The methodology was validated using a large-scale, real-world dataset from a commercial German-language dialog system. The dataset comprised 3.5 million training utterances and 560,000 test utterances across six domains (e.g., Music, Movies). Conflict rates varied by domain, ranging from 4.9% to 10.9%. To establish a ground truth, an expert linguist manually resolved a subset of 490 conflicts covering over 6,000 utterances.
Results and Discussion
The Max IG Cut proved highly effective, achieving up to 90% accuracy in identifying correct post-change labels at high IG thresholds. A key finding was that the LSTM model significantly outperformed majority voting in high-entropy cases, with its advantage peaking when annotator disagreement was total (NH = 1.0).
When the resolved dataset was used to train a production-grade NLU model, it yielded a 2.75% relative error reduction compared to a baseline without conflict resolution. This improvement was even more pronounced in specific contentious edge cases. Unlike methods that simply prune noisy data, this pipeline preserves the data distribution by replacing labels, which is critical for maintaining performance in production.
Conclusion and Future Work
This work demonstrates that treating time as a signal and using model-based resolution for high-disagreement cases can significantly improve NLU performance in evolving production environments. Future research will explore extending this architecture to non-linear labeling tasks (e.g., dependency parsing) and replacing the Bi-LSTM with Transformer-based architectures like BERT. Additionally, the authors plan to investigate multi-split approaches for datasets that have undergone multiple guideline iterations over several years.
Summary updated