TL;DR
This paper tackles the challenge of noisy Automatic Speech Recognition (ASR) transcripts degrading the performance of Spoken Language Understanding (SLU) models. They introduce a novel "confusion-aware" fine-tuning method for pre-trained language models (LMs) that explicitly teaches them to handle acoustically similar but incorrectly transcribed words, leading to more robust SLU systems.
What Problem Does This Paper Address?
Modern NLP heavily relies on contextualized word embeddings, often derived from large pre-trained language models (LMs). These models excel when fed clean text. However, in real-world spoken language applications, the input is often a transcript generated by an ASR system, which is prone to errors. These errors, especially those involving acoustically similar words (e.g., "fair" vs. "fare"), can significantly confuse SLU models, leading to incorrect intent detection and slot filling. Existing methods often train SLU models on simulated errors or use ASR outputs like lattices, but this paper focuses on making the embeddings themselves more robust to these ASR errors by leveraging the inherent acoustic confusability of words.
Methods
The core of the paper's contribution lies in a novel "confusion-aware fine-tuning" approach for pre-trained LMs. This method aims to make the LM's word representations more resilient to ASR mistakes. The process involves three main stages:
- Pre-training: A general-domain LM is pre-trained (using ELMo as the base architecture).
- Confusion-Aware Fine-tuning: This is the novel part. The pre-trained LM is fine-tuned on text from the target SLU task. Crucially, this fine-tuning incorporates a "confusion loss" that encourages the LM to produce similar embeddings for words that are acoustically confusable. This is achieved by identifying pairs of words that are often confused by ASR systems.
- Supervised Confusion Extraction: If both manual transcripts and ASR outputs are available, word confusion pairs are extracted by aligning them using an edit-distance criterion.
- Unsupervised Confusion Extraction: If only audio is available, an ASR system generates a word confusion network (WCN), and n-best hypotheses are aligned to extract confusions.
- SLU Model Training: A downstream SLU model (a biLSTM in this case) is trained using the fine-tuned, ASR-robust embeddings.
The joint objective function during fine-tuning balances the standard language modeling loss (LLM) with this new confusion loss (Lconf).
Datasets
The experiments were conducted on three standard SLU datasets:
- ATIS (Airline Travel Information Systems): A widely used benchmark for spoken language understanding, containing audio recordings for flight reservations.
- Snips SmartLights: Contains spoken commands for smart light assistants, with recordings in both close and far field settings.
- Snips: A larger dataset for benchmarking NLU systems, for which synthetic audio was generated using a text-to-speech system.
The paper also mentions using the WSJ dataset for training an ASR system for ATIS and Kaldi's pre-trained ASR models for other datasets.
Key Insights and Ideas
- The Embeddings Themselves Need Robustness: Instead of just adapting the SLU model, the paper argues for making the underlying word representations more robust to ASR errors. This is a more fundamental approach.
- Acoustic Confusability is Key: The core idea is to leverage the fact that humans can often disambiguate acoustically similar words. By explicitly teaching LMs about these confusions, their representations become more resilient.
- Unsupervised Adaptation is Powerful: The unsupervised confusion extraction method is particularly valuable as it doesn't require manual transcriptions, making it applicable in more scenarios.
- Fine-tuning LMs with Confusion Loss Works: The proposed confusion-aware fine-tuning significantly improves SLU performance on noisy ASR transcripts, outperforming baselines that don't account for these errors.
Key Results
The proposed confusion-aware fine-tuning method consistently improved SLU performance across all tested datasets when evaluated on ASR transcripts.
- For ATIS, the performance drop on ASR transcripts was significantly reduced compared to standard pre-trained ELMo embeddings.
- The method showed particular effectiveness on datasets with higher Word Error Rates (WER), like Snips.
- The confusion loss term (Lconf) provided a notable boost in performance beyond just language model fine-tuning (LLM). For example, on ATIS, the method using both LLM and Lconf (row (e) and (f) in Table 2) achieved higher accuracies on ASR transcripts than just LLM fine-tuning (row (d)).
Limitations and Future Work
- Limitations: The paper notes that the SLU datasets used are relatively small, which is a common challenge in SLU research. The unsupervised confusion extraction relies on the quality of the ASR system's WCN output.
- Future Work: The authors suggest that their method can be extended to other SLU tasks beyond intent detection and slot filling. Further exploration of different confusion extraction techniques and their impact on robustness could also be valuable. The paper also implies that this approach could be applied to other pre-trained models beyond ELMo.
Summary updated