Towards Reducing the Need for Speech Training Data to Build Spoken Language Understanding Systems

Samuel Thomas, Hong-Kwang Jeff Kuo, Brian Kingsbury, George Saon

TL;DR

Researchers at IBM have developed a way to build high-performing Spoken Language Understanding (SLU) systems using mostly text data, drastically reducing the need for expensive, human-annotated speech. By representing text as "textograms"—a frame-level format that mimics speech features—they created models that achieve 90% of full-performance with zero speech training, and 97% performance with just 10% of the usual speech data.

What Problem Does This Paper Address?

Building "end-to-end" (E2E) systems that understand spoken intent (e.g., "book a flight") usually requires thousands of hours of speech recordings paired with semantic labels. While raw text data is abundant, annotated speech is a bottleneck.

Previous attempts to bridge this gap often relied on Text-to-Speech (TTS) synthesis to "fake" speech data or complex multi-modal architectures that are difficult to train. The authors sought a simpler, more elegant solution: a single model architecture (RNN-Transducer) that can treat text and speech as essentially the same type of input, allowing it to learn from whichever is available.

Methods

The core innovation is the Textogram.

Standard speech models process "log-mel" features (visual representations of sound frequencies over time). To make text compatible, the authors convert words into "textograms"—2D representations where each character (grapheme) is represented as a 1-hot encoding stretched over a fixed number of time frames.

The Training Pipeline: 1. Pre-training: An RNN-Transducer (RNN-T) Automatic Speech Recognition (ASR) model is trained on both real speech and textograms. This teaches the "encoder" part of the network to produce similar embeddings regardless of whether the input is sound or text. 2. Robustness Tricks: To prevent the model from simply memorizing text, they use "Label Masking" (randomly dropping characters) and "Label Confusions" (simulating acoustic similarities, like swapping 'p' and 'b'). 3. Adaptation: The pre-trained model is then fine-tuned for SLU tasks (like identifying intents or entities). If only text is available, they freeze the encoder and only update the "prediction" and "joint" networks.

Datasets

The authors evaluated their approach on three distinct SLU benchmarks:

  • HarperValleyBank: A public dataset of simulated consumer banking dialogs (1,446 conversations).
  • Call Center Data: An internal IBM dataset of real-world customer utterances labeled with 29 different intent classes.
  • ATIS (Airline Travel Information System): A classic benchmark for both intent recognition and "slot filling" (entity extraction, like identifying "Boston" as a destination).

Key Insights and Ideas

  • Modal Agnosticism: By forcing text into a frame-level "speech-like" format, the RNN-T's encoder becomes a universal processor. This eliminates the need for a separate text-processing branch in the neural network.
  • The Power of the First 10%: The most striking insight is the diminishing returns of speech data. Adding just 1.7 hours of speech (10%) to a text-only model closed the performance gap significantly, suggesting that text provides the "knowledge" while a tiny bit of speech provides the "acoustic grounding."
  • Direct Text Usage: Unlike TTS-based methods, this approach doesn't require generating actual audio files, saving massive amounts of computation and storage during the training phase.

Key Results

  • HarperValleyBank: A model trained on zero speech achieved an F1 score of 45.05. Adding just 10% of the speech data boosted this to 53.84—nearly matching the 54.95 score achieved using 100% of the speech data.
  • Call Center Intent: The text-only model reached 76.97% accuracy. With 10% speech, it jumped to 88.34%, which is 99% of the performance of the full-speech model (89.06%).
  • ATIS Entity Recognition: The system achieved an F1 score of 85.95 with text only. Adding 10% speech data actually outperformed the model trained on all available speech, likely due to the data augmentation effect of the textograms.

Limitations and Future Work

The authors noted that while the system is highly effective, performance on the ATIS dataset was slightly more sensitive to speaker coverage. They hypothesize that more diverse speech data is still needed to handle high acoustic variability between different speakers.

Future research directions include exploring variable label durations (making some characters longer than others to better mimic natural speech rhythms) and testing the approach on more complex, multi-turn dialogue tasks.

Summary updated

Abstract

The lack of speech data annotated with labels required for spoken language understanding (SLU) is often a major hurdle in building end-to-end (E2E) systems that can directly process speech inputs. In contrast, large amounts of text data with suitable labels are usually available. In this paper, we propose a novel text representation and training methodology that allows E2E SLU systems to be effectively constructed using these text resources. With very limited amounts of additional speech, we show that these models can be further improved to perform at levels close to similar systems built on the full speech datasets. The efficacy of our proposed approach is demonstrated on both intent and entity tasks using three different SLU datasets. With text-only training, the proposed system achieves up to 90% of the performance possible with full speech training. With just an additional 10% of speech data, these models significantly improve further to 97% of full performance.
BibTeX
@inproceedings{thomas2022towards,
  author = {Samuel Thomas and Hong-Kwang Jeff Kuo and Brian Kingsbury and George Saon},
  title = {{Towards Reducing the Need for Speech Training Data to Build Spoken Language Understanding Systems}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=INj9uOlbhTo},
}

Loading related content...

Figure