Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding

Suyoung Kim, Jiyeon Hwang, Ho-Young JungNAACL 2024 · DOI

TL;DR

This paper introduces a novel two-stage learning method, Contrastive and Consistency Learning (CCL), to improve Spoken Language Understanding (SLU) models' robustness against Automatic Speech Recognition (ASR) errors. By aligning noisy and clean transcripts at both token and utterance levels and enforcing consistency in latent representations, CCL significantly enhances intent classification performance in challenging noisy environments.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) systems are crucial for enabling natural human-computer interaction. While end-to-end models have shown promise, they require vast amounts of labeled data and are sensitive to variations in speech input. Module-based approaches, which leverage pre-trained language models, offer an alternative but struggle with the inherent inconsistencies introduced by ASR errors (e.g., misspellings, insertions, deletions). These errors can lead to misinterpretations of user intent, degrading the overall performance of SLU systems. This paper addresses the critical challenge of making SLU models robust to these ASR-induced transcription errors, particularly when dealing with complex and diverse user queries.

Methods

The authors propose a two-stage approach called Contrastive and Consistency Learning (CCL) to enhance a neural noisy-channel model for SLU.

  • Stage 1: Token-based Contrastive Learning: This stage aims to correlate errors between clean and noisy ASR transcripts.
  • Selective-token Contrastive Learning: This novel technique aligns word tokens from clean and noisy transcripts by considering edit distance. It creates positive pairs from tokens with minimal edits and negative pairs from non-matching tokens, effectively learning to correlate ASR errors at the word level.
  • Utterance Contrastive Learning: This component aligns the overall meaning of the clean and noisy transcripts by treating them as a positive pair. It uses the [CLS] token representations to capture the implied meaning of the entire utterance.
  • The combined token-based contrastive objective ($L_{ctr}$) is a weighted sum of the selective-token and utterance contrastive losses.
  • Stage 2: Consistency Learning: This stage ensures that the latent features of the noisy ASR transcript remain coherent with those of the clean transcript. It uses a linear projection to map the encoder outputs and minimizes the distance between the noisy and referenced latent features, as well as their target probabilities. This helps the model predict the correct intent even when the noisy transcript contains significant errors.

The overall loss function combines the contrastive and consistency learning objectives. During inference, only the inference network is used, ensuring no increase in computational cost.

Datasets

The proposed CCL method was evaluated on four benchmark SLU datasets:

  • SLURP: A challenging dataset of house robot voice commands with 60 intent labels, known for its inaccurate labels.
  • Timers: A simpler dataset with four labels for smart-home or voice assistants.
  • FSC: A dataset with 31 intents for smart-home or voice assistants.
  • SNIPS: A well-known text-based intent classification dataset with seven intents.

The authors also utilized noisy ASR transcripts generated by Google API, HuBERT, and Wav2vec 2.0, as well as those provided in previous works (Chang and Chen, 2022; Kim et al., 2022).

Key Insights and Ideas

  • Token-level alignment is crucial for ASR error correction: Simply aligning entire utterances isn't enough; correlating errors at the word token level is essential for robust intent classification.
  • Consistency learning bridges the gap: Enforcing consistency between the latent representations of clean and noisy transcripts helps the model generalize better and avoid misclassifications caused by ASR errors.
  • Noisy-channel models can be effectively enhanced: The noisy-channel paradigm, when augmented with contrastive and consistency learning, proves highly effective in handling the complexities of ASR errors.
  • Pre-trained models still struggle with severe errors: Even advanced LLMs like GPT-4 can falter when ASR errors significantly distort the original meaning, highlighting the need for specialized training strategies.

Key Results

  • On the SLURP dataset, the CCL method achieved a significant improvement of 2.59% in accuracy and 2.48% in F1-score under severe ASR errors (Noisy0.56) compared to the Noisy-CE baseline.
  • CCL outperformed existing module-based and end-to-end approaches on the SNIPS dataset, achieving 98.72% accuracy.
  • The method demonstrated robustness across various ASR error rates and on different datasets, including complex ones like FSC.
  • Ablation studies showed that both token-based contrastive learning and consistency learning contribute significantly to the performance gains, with token-based contrastive learning being particularly effective in noisy environments.
  • Qualitative analysis, including t-SNE visualizations, showed that CCL effectively aligns clean and noisy transcripts at both utterance and word token levels, reducing the distance between them and improving intent prediction.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Severe ASR errors: When ASR errors completely obscure the original meaning, even CCL may struggle to predict the correct intent. Future work aims to develop methods that address the mismatch between noisy ASR transcripts and their corresponding labels.
  • Multilingual SLU: The current work focuses on English. Expanding the approach to handle ASR errors in multilingual SLU is a promising future direction.

Additionally, the paper notes that while encoder-based LLMs show promise, further research is needed to fully understand their capabilities in handling ASR errors without extensive fine-tuning.

Summary updated

BibTeX
@inproceedings{kim2024contrastive,
  author = {Suyoung Kim and Jiyeon Hwang and Ho-Young Jung},
  title = {{Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.318},
}
Figure