TL;DR
This paper tackles the challenge of building accurate Natural Language Understanding (NLU) systems for voice assistants in new domains where labeled training data is scarce. They propose combining three weakly supervised machine learning techniques to leverage readily available unlabeled user utterances, achieving significant improvements in semantic error rates.
What Problem Does This Paper Address?
Creating sophisticated voice-enabled applications for virtual assistants like Alexa is a growing trend. However, a major bottleneck for third-party developers is the difficulty and cost of acquiring sufficient labeled data to train robust NLU models. While large pre-trained language models have helped, they haven't entirely eliminated the need for domain-specific training data. This paper addresses this gap by exploring how to effectively use unlabeled, automatically transcribed user utterances to improve NLU performance in these low-resource scenarios, specifically for the joint tasks of Intent Classification (IC) and Named Entity Recognition (NER).
Methods
The authors investigate three distinct approaches to augment training data for NLU models:
- Maximal FST-Matching: This method leverages the developer-provided "carrier phrases" (templates for utterances) and their associated intent/slot labels. They construct a Finite State Transducer (FST) from these phrases. Then, they process live, unlabeled user utterances, finding the "maximal" sub-utterance that matches a carrier phrase. The intent and slot labels from the matched carrier phrase are then "superimposed" onto the entire utterance. This allows them to automatically label utterances that are not exact matches but share significant semantic overlap. They use a "span ratio" to filter out potentially noisy matches.
- Sentence Embeddings via Seq2Seq Auto-Encoder: This approach involves pre-training a sequence-to-sequence auto-encoder (s2s-AE) on large amounts of unlabeled, automatically transcribed user traffic. The encoder part of this s2s-AE is then used to generate sentence embeddings. These embeddings are injected as an additional signal into the baseline NLU model, providing a richer representation of the utterance's meaning. They explore both pre-training and fine-tuning this auto-encoder.
- Tri-training: This is a semi-supervised learning (SSL) technique that uses an ensemble of three independently trained classifiers. They train three NLU models on the initial labeled data and then iteratively use them to label unlabeled data. When all three classifiers agree on the label for an unlabeled utterance, that utterance is considered high-precision and added to the training set for the next iteration. This process is repeated until a stopping criterion is met.
Datasets
The study utilizes a dataset of live, automatically transcribed user traffic spanning several months across 86 third-party Alexa skills. This data is de-identified. For training, they use a baseline of ten thousand carrier phrases per skill, with adjustments for out-of-domain intents. For evaluation, they use in-house annotated test sets of live user traffic. They also mention using a large corpus for pre-training the auto-encoder.
Key Insights and Ideas
- Synergy is Key: The most significant improvements come from combining multiple weakly supervised techniques, suggesting that different methods capture complementary aspects of the unlabeled data.
- Leveraging Imperfect Matches: The maximal FST-matching approach highlights that even partial matches to developer-provided templates can yield valuable labeled data, provided they are filtered appropriately.
- Unsupervised Representation Learning Matters: Injecting sentence embeddings from a pre-trained auto-encoder provides a strong signal, even without skill-specific fine-tuning, demonstrating the power of unsupervised representation learning.
- High-Precision Labeling: Tri-training effectively identifies high-confidence labels from unlabeled data, significantly expanding the training set size.
Key Results
The paper reports significant relative improvements in Semantic Error Rate (SemER) and Interpretation Recognition Error Rate (IRER) compared to their baseline NLU model:
- Individual Methods:
- Maximal FST-Matching: 5.01% improvement on SemER and 1.44% on IRER.
- Tri-training: 7.65% improvement on SemER and 9.67% on IRER.
- Sentence Embedding Injection: 4.73% improvement on SemER.
- Combined Methods:
- Maximal FST-Matching + Tri-training: 10.54% improvement on SemER and 11.49% on IRER.
- Base+SE (Sentence Embedding) + Maximal FST-Matching + Tri-training: 11.77% improvement on SemER and 12.94% on IRER. This represents the best overall performance.
Limitations and Future Work
- Skill-Specific Thresholding: While global thresholds for FST-matching showed improvements, skill-specific thresholding yielded even better results, suggesting that domain-specific tuning is beneficial.
- SEMER Degradation: In some cases, particularly with tri-training, a small number of skills experienced SEMER degradation. This indicates that while these methods are generally effective, careful tuning and filtering are crucial to avoid introducing noise.
- Future Work: The authors suggest that these methods can be deployed over time into a cycle of continual learning, implying ongoing improvement as more unlabeled data becomes available. They also explored adding SVM classifiers to their tri-training ensemble, which showed promise but reduced coverage. Further investigation into combining different types of classifiers could be a future direction.
Summary updated