Learning Spoken Language Representations with Neural Lattice Language Modeling

Chao-Wei Huang, Yun-Nung Chen

TL;DR

This paper introduces a novel framework for learning representations from spoken language by extending pre-trained language models to handle "lattices" – structured outputs from speech recognition systems that capture ambiguity. Their two-stage pre-training approach, first on text and then on lattices, significantly improves performance on spoken language understanding tasks like intent detection and dialogue act recognition.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) typically involves two steps: Automatic Speech Recognition (ASR) to convert speech to text, followed by Natural Language Understanding (NLU) on the transcribed text. While pre-trained language models (like BERT, ELMO) have revolutionized NLU for written text, they struggle with spoken language due to the inherent ambiguities and errors introduced by ASR. Standard ASR outputs a single "best guess" transcription, discarding valuable information about alternative interpretations. This paper addresses the challenge of effectively leveraging the richer, ambiguous information present in ASR lattices to improve SLU performance.

Methods

The core contribution is a Neural Lattice Language Model (LLM). This model is designed to process ASR lattices, which are directed acyclic graphs (DAGs) representing multiple recognition hypotheses. The LLM uses a LatticeLSTM (a generalization of LSTMs to lattice structures) as its backbone.

The key innovation lies in a two-stage pre-training strategy:

  1. Stage 1: Pre-train on Sequential Texts: A bidirectional LSTM language model is pre-trained on a large general-domain text corpus, similar to existing approaches like ELMO. This establishes a strong foundation for understanding language structure.
  2. Stage 2: Pre-train on Lattices: The weights from the Stage 1 LSTM are used to initialize a bidirectional LatticeLSTM. This LatticeLSTM is then further pre-trained on lattices from the target task's training data using a novel lattice language modeling objective. This objective aims to predict the next transition (word) within a lattice given the preceding context represented by the lattice structure.

This two-stage approach is motivated by the difficulty of obtaining large amounts of lattice data and the computational inefficiency of LatticeLSTMs compared to standard LSTMs.

Datasets

The paper evaluates its framework on benchmark datasets for spoken language understanding:

  • Intent Detection: ATIS (Airline Travel Information Systems) and SNIPS (Service-based Intent Recognition and Natural Language Understanding).
  • Dialogue Act Recognition: Switchboard Dialogue Act Corpus (SWDA) and ICSI Meeting Recorder Dialogue Act Corpus (MRDA).

For SNIPS, which is originally text-based, the authors synthesized a spoken version using a text-to-speech service. ASR systems were used to transcribe the datasets, and the performance was evaluated using both manual transcripts and ASR outputs (including lattices).

Key Insights and Ideas

  • Lattices are Richer than 1-Best Transcripts: The paper strongly demonstrates that incorporating information from ASR lattices, rather than just the single best transcription, leads to significant performance gains in SLU tasks.
  • Two-Stage Pre-training is Effective and Efficient: The proposed two-stage pre-training method successfully transfers knowledge from general text to spoken language lattices, reducing the need for extensive lattice-specific training data and improving efficiency.
  • Generalization of Sequential LMs: The work highlights that sequential LMs are a special case of lattice LMs, enabling the effective transfer of weights from a pre-trained sequential LM to a lattice LM.
  • Contextualized Lattice Representations: The LLM learns to produce contextualized embeddings for nodes within the lattice, capturing the nuances of spoken utterances and their ambiguities.

Key Results

The proposed method consistently outperforms strong baselines on intent detection and dialogue act recognition tasks when evaluated on spoken inputs (ASR outputs).

  • Significant Improvements: The paper reports relative error reductions ranging from 3% to 42% compared to a biLSTM with ELMO embeddings when using ASR 1-best outputs.
  • Outperforming Baselines: The proposed method achieves the best results on most datasets, outperforming baselines like BERT-base (when using ASR 1-best) and a standard biLatticeLSTM.
  • Ablation Studies: Skipping either pre-training stage (Stage 1 or Stage 2) leads to degraded performance, confirming the importance of both stages.
  • Lattice Input Benefit: Even when evaluating the proposed model on 1-best results (after fine-tuning), it still shows benefits from using lattice inputs during training.

Limitations and Future Work

  • Computational Cost of LatticeLSTMs: The paper acknowledges that LatticeLSTMs are generally less efficient than standard RNNs due to difficulties in parallelization.
  • Data Requirements: While the two-stage approach reduces data demands, obtaining sufficient high-quality lattices for the second stage of pre-training remains a consideration.
  • Future Work: The authors suggest that their framework is capable of providing high-quality representations of lattices, implying potential for further exploration in other spoken language processing tasks. The availability of the code (https://github.com/MiuLab/Lattice-ELMo) also facilitates future research.

Summary updated

Abstract

Pre-trained language models have achieved huge improvement on many NLP tasks. However, these methods are usually designed for written text, so they do not consider the properties of spoken language. Therefore, this paper aims at generalizing the idea of language model pre-training to lattices generated by recognition systems. We propose a framework that trains neural lattice language models to provide contextualized representations for spoken language understanding tasks. The proposed two-stage pre-training approach reduces the demands of speech data and has better efficiency. Experiments on intent detection and dialogue act recognition datasets demonstrate that our proposed method consistently outperforms strong baselines when evaluated on spoken inputs. The code is available at https://github.com/MiuLab/Lattice-ELMo.
BibTeX
@inproceedings{huang2020learning,
  author = {Chao-Wei Huang and Yun-Nung Chen},
  title = {{Learning Spoken Language Representations with Neural Lattice Language Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=TklLgrjXWSr},
}
Figure