Learning Spoken Language Representations with Neural Lattice Language Modeling

Chao-Wei Huang, Yun-Nung Chen

Abstract: Chao-Wei Huang, Yun-Nung Chen. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

Introduction: The Challenge of Noisy Speech

Most modern Spoken Language Understanding (SLU) systems operate in a pipeline: an Automatic Speech Recognition (ASR) engine converts audio to text, which is then processed by a Natural Language Processing (NLP) model to extract intent or dialogue acts. However, this approach suffers from error propagation. If the ASR engine selects an incorrect "1-best" transcription, the downstream NLP model—often a powerful pre-trained transformer like BERT—has no way to recover the original meaning.

ASR engines actually produce more than just a single string; they generate lattices. A lattice is a Directed Acyclic Graph (DAG) that encodes multiple transcription hypotheses and their relative probabilities. While these graphs contain the "correct" transcription more often than the 1-best output, most modern language models are designed for linear text and cannot natively process graph structures.

Example of an ASR lattice representing multiple transcription hypotheses with associated probabilities.

Proposed Method: Neural Lattice Language Modeling

To bridge this gap, the authors introduce a framework for pre-training language models directly on ASR lattices. The core of their approach is the LatticeRNN (specifically implemented as a LatticeLSTM), which generalizes standard recurrent units to operate on graph nodes. In a LatticeRNN, the hidden state of a node is computed by pooling the hidden states of all incoming transitions, weighted by their ASR confidence scores.

The training process follows a strategic two-stage pre-training pipeline:

  1. Stage 1: Pre-Training on Sequential Text. The model (a bidirectional LSTM) is first trained on a massive corpus of general written text. This allows it to capture fundamental linguistic patterns, syntax, and semantics from clean, high-quality data.
  2. Stage 2: Pre-Training on Lattices. The model is "upgraded" to a LatticeLSTM and continues pre-training on lattices generated from the target task's training data. This stage adapts the model to the specific structural noise and alternative hypotheses present in spoken language.

The model is trained using a Lattice Language Modeling objective, where it learns to predict the next word in the lattice given the preceding sub-lattice.

The proposed two-stage pre-training framework: from sequential text to lattice-based representations for downstream classification.

Experimental Setup and Datasets

The researchers evaluated the effectiveness of lattice-based representations across four benchmark SLU datasets:

  • ATIS (Airline Travel Information Systems): Spoken queries regarding flight information.
  • SNIPS: A voice assistant dataset (speech synthesized from text to create lattices).
  • SWDA (Switchboard Dialogue Act): Recorded telephone conversations.
  • MRDA (ICSI Meeting Recorder Dialogue Act): Multi-party meeting transcripts and audio.

Key Insights

  • Lattices as Generalizations: A standard sentence is essentially a linear-chain lattice. This conceptual link allows the authors to initialize the complex LatticeRNN with weights from a model pre-trained on simple text, making the training process much more efficient.
  • Ambiguity Preservation: By processing the entire lattice, the model can attend to alternative paths that may make more semantic sense than the top ASR hypothesis, effectively "recovering" from ASR errors.
  • Efficiency through Staging: Training LatticeRNNs from scratch on massive datasets is computationally difficult due to parallelization constraints. The two-stage approach provides a practical path to high-performance spoken language representations.

Results and Performance

The proposed method consistently outperformed strong text-based baselines:

  • Error Reduction: The model achieved relative error reductions of 3% to 42% compared to standard sequential language models.
  • Comparison to BERT: On the ATIS dataset, the lattice-based model reached 95.84% accuracy, rivaling BERT-base (95.97%) even though BERT was trained on significantly more parameters and data. This suggests that structural awareness of ASR noise can compensate for raw model scale.
  • Ablation Studies: Results confirmed that both stages of pre-training are essential; skipping the text-based stage loses general linguistic knowledge, while skipping the lattice stage prevents the model from learning how to handle ASR-specific graph structures.

Conclusion and Future Directions

This work demonstrates that pre-training on lattices is a powerful way to build robust SLU systems. While LatticeRNNs are harder to parallelize than Transformers, they offer a significant advantage in handling the inherent ambiguity of speech. Future research may look into "Lattice Transformers" to improve scalability or explore tighter integration with end-to-end speech models that could potentially generate these representations without an explicit lattice generation step.

Summary updated

BibTeX
@inproceedings{huang2020learning,
  author = {Chao-Wei Huang and Yun-Nung Chen},
  title = {{Learning Spoken Language Representations with Neural Lattice Language Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=TklLgrjXWSr},
}
Figure