TL;DR
This paper introduces a "human verification" workflow for training Natural Language Understanding (NLU) models. By having humans review pre-computed NLU hypotheses instead of annotating from scratch, they achieve significant cost reductions (up to 60%) and improved model performance, demonstrating that verification is more efficient than annotation for mature NLU systems.
What Problem Does This Paper Address?
Developing and maintaining robust Natural Language Understanding (NLU) models, like those powering virtual assistants, requires continuous updates with new training data. The traditional method of acquiring this data involves manual annotation of user utterances, which is a costly and time-consuming process. As NLU models become more sophisticated, the volume of data needed for retraining grows, exacerbating the cost challenge. This paper addresses the need for a more efficient and cost-effective approach to data acquisition for NLU model updates, aiming to maintain or improve model performance while drastically reducing manual annotation efforts.
Methods
The core of this paper is the proposed "human verification workflow," which contrasts with the standard "human annotation workflow."
- Human Annotation Workflow (Benchmark): In this traditional approach, new utterances are manually annotated by humans from scratch for each model update. Annotators only have access to annotation guidelines.
- Human Verification Workflow (Proposed): This novel approach leverages the existing NLU model. Before a model update, the previous version of the NLU model generates "hypotheses" (predicted annotations) for new utterances. Human annotators then verify these hypotheses. If the hypothesis is correct, it's used as ground truth. If incorrect, the utterance is then sent for full manual annotation. This means humans only annotate utterances where the model's prediction is wrong. Crucially, verifiers have access to both the annotation guidelines and the model's hypothesis.
The authors also incorporate Active Learning (AL). Recognizing that the verification workflow frees up annotation resources, they suggest using these savings to apply AL to select the most impactful utterances for annotation, further boosting model performance.
Datasets
The experiments were conducted using a large dataset of unlabelled utterances representative of user engagement with a dialog system. This dataset spanned a wide range of intents and slots.
- Training Set: 400,000 utterances.
- Test Set: 100,000 utterances.
The authors emphasize that for their evaluation, they simulated a "mature" NLU system where the training data is derived entirely from either the verification or annotation workflow, not a mixed set.
Key Insights and Ideas
- Verification is Faster and Cheaper: The fundamental idea is that verifying a pre-existing hypothesis is significantly quicker and less cognitively demanding than creating an annotation from scratch.
- Human Error Reduction: The verification workflow leads to fewer human errors in annotation. This is attributed to the fact that having the NLU hypothesis provides a strong indication, reducing ambiguity and the likelihood of misinterpretation compared to annotating from a blank slate.
- Consistency Boost: The verification workflow results in more consistent training data, as evidenced by lower entropy values. This suggests that the model receives clearer and less conflicting signals during training.
- Synergy with Active Learning: The cost savings from the verification workflow can be effectively reinvested into Active Learning, leading to substantial performance gains with minimal increases in annotation volume.
Key Results
The proposed human verification workflow demonstrated significant improvements:
- Annotation Volume Reduction: Up to 97% reduction in manual annotation volume.
- Cost Reduction: Approximately 60% overall cost reduction per iteration.
- Model Performance Improvement:
- Up to 14.16% improvement on a recall-based metric (SemER).
- Up to 9.57% improvement on an F1-score based metric (F-SemER).
- These improvements were observed for both CRF+MaxEnt and BERT architectures.
- Human Error Reduction: Overall human errors were reduced by 66% compared to the annotation workflow, with intent errors seeing an 80% reduction.
- Active Learning Impact: Adding a small amount of AL-annotated data (30,000 utterances) to the verification workflow resulted in performance gains exceeding 20% for most metrics.
Limitations and Future Work
- Maturity Assumption: The study assumes a "mature" NLU system where a previous model version is available to generate hypotheses. The initial stages of NLU development, where no prior model exists, would still rely on full annotation.
- Verification Errors: While reduced, verification errors can still occur. The paper acknowledges that the verification workflow might ingest incorrect training data, but argues that the overall impact is positive due to the significant reduction in annotation volume and improved consistency.
- Future Work: The authors suggest that the freed-up annotation resources from the verification workflow can be strategically used for other impactful tasks, such as Active Learning, which they demonstrate yields significant performance benefits. They also imply that further exploration into how to best utilize these savings could be beneficial.
Summary updated