Abstract
Compared to hybrid automatic speech recognition (ASR) systems that use a modular architecture in which each component can be independently adapted to a new domain, recent end-to-end (E2E) ASR system are harder to customize due to their all-neural monolithic construction. In this paper, we propose a novel text representation and training framework for E2E ASR models. With this approach, we show that a trained RNN Transducer (RNN-T) model's internal LM component can be effectively adapted with text-only data. An RNN-T model trained using both speech and text inputs improves over a baseline model trained on just speech with close to 13% word error rate (WER) reduction on the Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation. The usefulness of the proposed approach is further demonstrated by customizing this general purpose RNN-T model to three separate datasets. We observe 20-45% relative word error rate (WER) reduction in these settings with this novel LM style customization technique using only unpaired text data from the new domains.
Introduction
Modern Automatic Speech Recognition (ASR) has seen a significant shift from modular "hybrid" systems to monolithic End-to-End (E2E) architectures like the Recurrent Neural Network Transducer (RNN-T). While E2E models offer superior accuracy and simplified deployment, they present a significant challenge: domain adaptation. Hybrid systems allow for independent updates to the language model (LM) using text-only data. In contrast, E2E models typically require paired audio and transcriptions, which are often scarce in specialized domains like banking or medical services.
This work introduces a novel framework to bridge this gap by enabling E2E models to learn directly from text-only data. The core of this approach is the "textogram," a 2D representation of text that mimics the structure of acoustic features.
The Textogram Representation
To allow an RNN-T model to process text similarly to speech, the authors developed the textogram. Instead of using standard embeddings or one-hot vectors, text is converted into a pseudo-spectrogram format.
As shown in Figure 3, a textogram is constructed by taking the one-hot encoding of a character and repeating it over a fixed duration (e.g., 4 frames). This results in a 2D feature map that shares the same time-frequency structure as log-Mel spectrograms used for audio. This alignment allows the model's encoder to process both modalities through a shared interface, forcing the network to produce similar internal embeddings regardless of whether the input is audio or text.
Multi-Modal Training and Adaptation
The proposed TOG-RNN-T (Textogram-enhanced RNN-T) utilizes a multi-modal encoder. During training, the model is presented with both speech and text samples. For speech inputs, the textogram features are zeroed out; for text inputs, the speech features are zeroed out. To ensure robustness and prevent memorization, the authors employ label masking—randomly dropping characters in the textogram—to mimic the variability of human speech.
When moving to a new domain (like a banking app), the adaptation process involves freezing the encoder and updating only the "prediction" and "joint" networks using domain-specific textograms. This "holistic" adaptation ensures the model remains grounded in the encoder's learned representations while specializing its linguistic knowledge, avoiding the need for complex "Shallow Fusion" during inference.
Experimental Results
The TOG-RNN-T approach was evaluated on both general telephony data and specific domain adaptation tasks, showing consistent improvements over speech-only baselines.
General Performance
On the NIST Hub5 2000 evaluation set, the TOG-RNN-T model achieved significant gains. By treating transcripts of the speech data as additional textogram inputs, the model effectively doubled its training exposure. This led to a Word Error Rate (WER) reduction from 6.9% to 6.2% on the Switchboard (SWB) set and from 11.9% to 10.5% on CallHome (CH).
Domain Adaptation
The framework's primary strength lies in its ability to adapt to new domains using only text. The authors tested this on three benchmarks: SLURP (home/office), ATIS (airline travel), and Harper Valley Bank (banking).
As seen in Figure 6, for the Harper Valley Bank dataset, the unadapted model started at 10.4% WER. Applying the textogram adaptation (TOG adapt) significantly reduced the error rate to 6.8%, outperforming standard neural LM adaptation techniques. Across all datasets, the relative WER reduction ranged from 20% to 45%. Notably, the ATIS dataset saw a reduction from 3.1% to 1.7% WER.
Conclusion
The textogram framework provides a powerful method for injecting text-only knowledge into monolithic E2E ASR models. By aligning text and speech features in a shared 2D space, the TOG-RNN-T model achieves state-of-the-art adaptation performance without the need for synthetic audio or complex decoding-time fusion. This approach simplifies the inference engine and reduces latency while maintaining high accuracy in specialized domains.
Summary updated