TL;DR
This paper tackles the challenge of efficiently expanding natural language understanding (NLU) capabilities in dialogue systems by leveraging semi-supervised learning (SSL) with a focus on optimizing the diversity of augmented training data. They demonstrate that by strategically selecting diverse, automatically generated utterances, they can achieve comparable NLU performance with significantly less training data, leading to faster development cycles and more agile systems.
What Problem Does This Paper Address?
Developing new functionalities for task-oriented dialogue systems is crucial, but it's a costly process. Building accurate NLU models for these new features requires collecting and manually annotating large amounts of data. Semi-supervised learning (SSL) offers a solution by utilizing unlabeled data alongside existing labeled data, but simply adding all available unlabeled data can lead to diminishing returns and significantly increase training times. The core problem this paper addresses is how to make SSL for NLU more efficient by intelligently selecting which unlabeled data to use, specifically by optimizing for diversity within the augmented dataset. This is particularly important in production environments where new functionalities are constantly being introduced, leading to shifts in usage patterns and the need for rapid adaptation.
Methods
The authors propose a functionality-specific self-training approach for SSL. The process involves:
- Functionality Filtering: An initial classifier identifies utterances relevant to the target functionality from a pool of unlabeled data.
- Intent and NER Scoring: A fused system score, combining confidence from intent classification (IC) and named entity recognition (NER), is used to select candidate utterances. They aggregate data across various confidence thresholds, weighting more confident utterances higher.
- Diversity Selection: This is the key novel contribution. After initial selection, a diversity-driven approach is employed to sub-sample the augmented data. They explore two main techniques:
- Paraphrase Detection: They train a paraphrase detection model to identify utterance variants. This model uses a word-averaging embedding layer and combines embeddings, differences, and products of utterance pairs to predict paraphrase likelihood.
- Greedy Diversity Algorithm: This algorithm iteratively selects utterances from the augmented set that are most dissimilar to the existing annotated data (and previously selected augmented data). This aims to "fill in the gaps" of the current training distribution. They also compare this to sub-modular optimization and random selection.
Datasets
- Training Data: The authors used live traffic utterances from January 2017 to June 2018 as the pool for unlabeled SSL utterances. They also utilized the PPDB-S dataset (1.5 million paraphrase pairs) for training their paraphrase embedding models.
- Evaluation Data: They evaluated their approach on five NLU functionalities with varying complexity (number of intents and slots). The paper provides statistics on the number of annotated and test utterances for these functionalities.
Key Insights and Ideas
- Functionality-Specific SSL is Powerful: The paper strongly demonstrates that focusing SSL on individual functionalities, rather than a general system-wide approach, yields significant performance improvements.
- Diversity is Key to Efficiency: Simply augmenting with all available unlabeled data is not optimal. Prioritizing diverse utterances, which introduce lexical and syntactic variety, allows the model to learn more robustly with less data.
- Paraphrase Detection as a Diversity Metric: Using paraphrase detection to identify utterance variants is an effective way to ensure the augmented data covers a wider range of expressions for a given functionality.
- Greedy Diversity Algorithm Works: Their proposed greedy algorithm for selecting diverse utterances, prioritizing those dissimilar to existing data, proved to be a strong performer.
- Significant Data Reduction Possible: The most striking insight is that by employing diversity optimization, they can reduce the amount of augmented training data by up to 50% with minimal impact on NLU performance.
Key Results
- Functionality-specific self-training substantially improved production system performance, reaching up to 25% relative performance difference.
- Methods optimizing diversity (specifically their paraphrase-based selection) allowed for training data reduction of up to 50% while maintaining comparable performance to using unrestricted augmentation.
- In their benchmark experiments, selecting half of the augmentation data using their paraphrase detection model resulted in SER scores that were either equivalent or only slightly worse (1.34% relative decrease to 3.61% relative increase) compared to using all augmentation data.
- For 7 out of 25 experiments, training on 50% of the augmentation data with their paraphrase-based selection was statistically equivalent (at 95% confidence) to training on the full augmentation set.
Limitations and Future Work
- Limitations: The paper acknowledges that their model performance is not deterministic due to parallel computing and randomization, hence they report averaged performance over five training runs. The diversity selection methods are applied to sub-sample the already selected augmented data, implying a multi-stage filtering process.
- Future Work: The authors suggest future research directions including:
- Applying diversity-measure-based utterance clustering.
- Developing paraphrase-based strategies for utterance generation and augmentation.
- Investigating how their suggested SSL approach performs on state-of-the-art NLU models.
Summary updated