Integrating Text Inputs for Training and Adapting RNN Transducer ASR Models

Samuel Thomas, Brian Kingsbury, George Saon, Hong-Kwang Jeff Kuo

TL;DR

Researchers at IBM have developed a way to train and adapt "monolithic" End-to-End (E2E) speech recognition models using only text data. By representing text as "textograms"—a 2D format that mimics the structure of speech features—they achieved up to a 45% relative reduction in error rates when adapting models to new domains like banking or home assistants.

What Problem Does This Paper Address?

Traditional "hybrid" Automatic Speech Recognition (ASR) systems are modular, consisting of separate acoustic, lexicon, and language models. This makes them easy to customize: if you want to recognize medical jargon, you simply swap in a medical language model.

Modern End-to-End (E2E) models, like the RNN Transducer (RNN-T), are more accurate and simpler to deploy but are notoriously difficult to adapt. Because they are "monolithic" (all-neural), they typically require paired audio and transcript data for training. In the real world, we often have mountains of domain-specific text (e.g., chat logs, manuals) but very little corresponding audio. This paper tackles the challenge of how to effectively inject that "text-only" knowledge into an E2E model without needing to synthesize artificial audio.

Methods

The authors introduce a novel representation called a textogram.

  1. The Textogram: Instead of feeding raw text into the model, they convert it into a 2D representation. They take the one-hot encoding of a character and repeat it for a fixed duration (e.g., 4 frames). This creates a "pseudo-spectrogram" that has a similar time-frequency structure to actual speech features (log-Mel spectrograms).
  2. Multi-Modal Encoder: They modify the RNN-T's "transcription network" (the encoder) to accept a larger input vector—the sum of the dimensions of speech features and textogram features.
  3. Joint Training: During training, the model sees both modalities. For a speech sample, the textogram input is set to zero; for a text sample, the speech input is set to zero. This forces the encoder to produce similar internal embeddings regardless of whether the input was audio or a textogram.
  4. Robustness Techniques: To prevent the model from simply memorizing text, they use label masking (randomly dropping characters in the textogram) and suggest other variations like "label confusions" to mimic human speech errors.
  5. Adaptation: When moving to a new domain (like a banking app), they freeze the encoder and only update the "prediction" and "joint" networks using textograms from the new domain.

Datasets

  • Base Training: A massive collection of US English telephony data, including Switchboard, Fisher, and IBM proprietary data.
  • General Evaluation: The NIST Hub5 2000 evaluation set (Switchboard and CallHome).
  • Adaptation Benchmarks:
    • SLURP: A recent dataset for spoken language understanding in home/office environments.
  • ATIS: Airline Travel Information System (highly specific vocabulary).
  • Harper Valley Bank: A public domain corpus of simulated consumer banking dialogs.

Key Insights and Ideas

  • Text as Data Augmentation: The authors hypothesize that training on both modalities acts as a powerful regularizer. By using the transcripts of the speech data as additional textogram inputs, they effectively doubled their training data, leading to a 10-13% Word Error Rate (WER) reduction even on general tasks.
  • Holistic Adaptation: Unlike previous methods that treat the RNN-T's prediction network as a standalone Language Model (LM), the textogram approach adapts the network through the Joint Network. This ensures the adaptation is "aware" of how the encoder's embeddings behave, leading to better performance than traditional LM-only adaptation.
  • Simplicity in Decoding: Because the knowledge is baked directly into the RNN-T weights, there is no need for "Shallow Fusion" (combining an external LM during search), which simplifies the inference engine and reduces latency.

Key Results

  • Baseline Improvement: The "TOG-RNN-T" (Textogram-enhanced) model reduced WER from 6.9% to 6.2% on Switchboard compared to a standard speech-only baseline.
  • Domain Adaptation:
    • SLURP: Reduced WER from 47.5% (unadapted) to 36.5% (adapted).
  • ATIS: Reduced WER from 3.1% to 1.7% (a 45% relative improvement).
  • Harper Valley: Reduced WER from 10.4% to 6.8%.
  • Comparison: The textogram method consistently outperformed "Shallow Fusion" and other neural LM adaptation techniques across all tested domains.

Limitations and Future Work

  • Fixed Durations: Currently, the textogram uses a fixed duration of 4 frames per character. The authors acknowledge that real speech has variable durations and suggest that "Variable label duration" is a promising area for future study.
  • Phonetic vs. Graphemic: The study focused on characters (graphemes). Exploring how this works with phonetic sets or sub-word units (like WordPieces) remains to be seen.
  • Acoustic Realism: While label masking helps, the textograms are still "cleaner" than real audio. Incorporating "label confusions" (e.g., swapping 'p' and 'b' in the textogram) could further bridge the gap between text and noisy speech.

Summary updated

Abstract

Compared to hybrid automatic speech recognition (ASR) systems that use a modular architecture in which each component can be in-dependently adapted to a new domain, recent end-to-end (E2E) ASR system are harder to customize due to their all-neural monolithic construction. In this paper, we propose a novel text representation and training framework for E2E ASR models. With this approach, we show that a trained RNN Transducer (RNN-T) model’s internal LM component can be effectively adapted with text-only data. An RNN-T model trained using both speech and text inputs improves over a baseline model trained on just speech with close to 13% word error rate (WER) reduction on the Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation. The usefulness of the proposed approach is further demonstrated by customizing this general purpose RNN-T model to three separate datasets. We observe 20-45% relative word error rate (WER) reduction in these settings with this novel LM style customization technique using only unpaired text data from the new domains.
BibTeX
@inproceedings{thomas2022integrating,
  author = {Samuel Thomas and Brian Kingsbury and George Saon and Hong-Kwang Jeff Kuo},
  title = {{Integrating Text Inputs for Training and Adapting RNN Transducer ASR Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=yCRbBFz0BY},
}

Loading related content...

Figure