Abstract
While there have been several contributions exploring state of the art techniques for text normalization, the problem of inverse text normalization (ITN) remains relatively unexplored. The best known approaches leverage finite state transducer (FST) based models which rely on manually curated rules and are hence not scalable. We propose an efficient and robust neural solution for ITN leveraging transformer based seq2seq models and FST-based text normalization techniques for data preparation. We show that this can be easily extended to other languages without the need for a linguistic expert to manually curate them. We then present a hybrid framework for integrating Neural ITN with an FST to overcome common recoverable errors in production environments. Our empirical evaluations show that the proposed solution minimizes incorrect perturbations (insertions, deletions and substitutions) to ASR output and maintains high quality even on out of domain data. A transformer based model infused with pretraining consistently achieves a lower WER across several datasets and is able to outperform baselines on English, Spanish, German and Italian datasets.
Introduction
When users interact with voice assistants, the Automatic Speech Recognition (ASR) system generates a raw stream of words. However, for readability and downstream processing, this spoken output must be converted into a formatted written version—a process known as Inverse Text Normalization (ITN). For example, a spoken phrase like "twenty one oh five" should be rendered as "2105" or "October 20, 2020" depending on the context.
Historically, ITN has relied on Finite State Transducers (FSTs), which are complex systems of hand-crafted rules. While precise, FSTs are difficult to scale because they require linguistic experts to manually curate rules for every new language or domain. Furthermore, they often struggle with the inherent ambiguity and noise of conversational speech.
Methodology
To address the limitations of rule-based systems, we developed a scalable neural-based approach to ITN. Our architecture centers on a Transformer-based sequence-to-sequence (Seq2Seq) model, enhanced with several key components:
- Copy Attention: This mechanism allows the model to selectively "point" to and copy words from the input directly to the output. This is crucial for maintaining the integrity of words that should not be normalized, such as names or general nouns, thereby preventing "hallucinations."
- BERT-Fusion: We integrate features from a pre-trained BERT model into the Transformer’s encoder and decoder layers. This provides the model with deeper linguistic context, helping it disambiguate tricky cases like "Dr." (Doctor vs. Drive).
- Hybrid Framework: To ensure production-grade reliability, we implement a hybrid system where the neural model acts as the primary processor. If the model's confidence score is low, the system falls back to a traditional FST to ensure a sane output.
Data Preparation
Because high-quality ITN datasets are scarce, we utilized a synthetic data pipeline. By taking written text from public sources and using a Text-to-Speech (TTS) frontend to "reverse-engineer" spoken versions, we created a robust training set across multiple domains (Wikipedia, News, TED talks) and languages (English, Spanish, German, and Italian).
Evaluation Metrics
Standard Word Error Rate (WER) can be misleading for ITN. We introduced two specialized metrics to better evaluate performance:
- I-WER (Induced WER): Measures the error rate specifically on words that should be normalized.
- NI-WER (Non-Induced WER): Measures the error rate on words that should not be changed. A low NI-WER is essential to ensure the model does not accidentally alter names or unrelated text.
Results
Our empirical evaluations demonstrate that the Transformer-based neural approach significantly outperforms traditional FST baselines. On Wikipedia data, the subword-based RNN model improved WER by 84.7% over the FST. The Transformer model with BERT-fusion achieved the highest accuracy, reducing WER to just 0.9%.
Key findings from our experiments include:
- The Power of Copying: The Copy Attention mechanism was the single most effective tool for keeping NI-WER low, effectively enforcing a "do no harm" policy on non-ITN text.
- Multilingual Synergy: Training a single model on multiple languages improved performance for each individual language, suggesting that the model learns shared patterns in how numbers and dates are structured across cultures.
- Robustness to Noise: In real-world conversational data (the TEL dataset), the hybrid model maintained superior performance across all categories, including numbers, units, and dates.
- BART Limitations: Interestingly, general-purpose pre-trained models like BART performed poorly on ITN tasks, often failing on simple entities and inserting extra words, highlighting the need for specialized architectures like our BERT-fused Transformer.
Conclusion
By combining the flexibility of neural networks with the reliability of FSTs and the precision of copy mechanisms, we have created a robust, scalable solution for Inverse Text Normalization. This approach reduces the need for manual rule curation and provides a path forward for supporting high-quality ASR formatting across a wide array of languages and domains.
Summary updated