TL;DR
Researchers at Amazon AWS AI have developed a scalable, neural-based approach to Inverse Text Normalization (ITN)—the process of converting spoken ASR output (e.g., "twenty one oh five") into formatted written text ("2105" or "October 20, 2020"). By combining Transformer-based models with a "copy mechanism" and a safety-net Finite State Transducer (FST), they achieved significantly lower error rates across multiple languages and domains compared to traditional rule-based systems.
What Problem Does This Paper Address?
When you talk to a voice assistant, the speech-to-text system (ASR) produces a raw stream of words. However, humans prefer reading "it costs $5.00" rather than "it costs five dollars." This conversion is the job of Inverse Text Normalization (ITN).
Traditionally, ITN relies on Finite State Transducers (FSTs)—complex, hand-crafted sets of rules. While precise, FSTs are a nightmare to scale: they require linguistic experts to write thousands of rules for every new language or domain. Furthermore, FSTs struggle with the messy, ambiguous nature of conversational speech. This paper seeks to replace or augment these rigid rules with flexible neural networks that can learn from data.
Methods
The authors propose a multi-layered neural strategy:
- Transformer-based Seq2Seq: They use a standard Transformer architecture to translate spoken-form sequences into written-form sequences.
- Copy Attention: A critical addition that allows the model to "point" to words in the input and copy them directly to the output. This prevents the model from "hallucinating" or changing words that shouldn't be normalized (like names or general nouns).
- BERT-Fusion: To give the model better context, they extract features from a pre-trained BERT model and fuse them into the Transformer’s encoder and decoder layers.
- Hybrid Framework: Recognizing that neural models can still make "silly" mistakes, they propose a hybrid system. The neural model processes the text first and provides a confidence score; if the score is low or if specific errors are detected, a traditional FST acts as a fallback to ensure the output remains sane.
Datasets
Because high-quality ITN datasets are scarce, the authors built a synthetic data pipeline. They took written text from public sources and used a Text-to-Speech (TTS) frontend to "reverse-engineer" what the spoken version would sound like.
The models were trained and tested on:
- Wikipedia: Subsampled parallel written-spoken text.
- Multi-domain data: ParaCrawl (web crawl), News-C (news), MuST-C (TED talks), and CNN-DailyMail.
- Multilingual sets: Data for German, Spanish, and Italian.
- ASR Outputs: Real-world broadcast (BCN) and conversational (TEL) transcripts to test robustness against noisy input.
Key Insights and Ideas
- The "Safety" Metric: The authors introduced two new metrics: I-WER (error rate on words that should be normalized) and NI-WER (error rate on words that should not be changed). This is vital because a neural ITN model that correctly formats a date but accidentally changes "John" to "Jon" is a failure in production.
- Copying is King: The "Copy Attention" mechanism was the most effective way to keep NI-WER low. It enforces a "do no harm" policy on the non-ITN parts of the sentence.
- Pre-training Power: Fusing BERT representations significantly improved performance on out-of-domain data. It seems the broad linguistic knowledge in BERT helps the model disambiguate tricky cases (e.g., knowing when "Dr." means "Doctor" vs. "Drive").
- Multilingual Synergy: Training a single model on multiple languages (English, Spanish, German, Italian) actually improved performance for each individual language, likely due to shared lexical patterns in how numbers and dates are handled.
Key Results
- Massive Gains over Baselines: On Wikipedia data, the subword-based RNN model improved Word Error Rate (WER) by 84.7% compared to the traditional FST baseline.
- Transformer Superiority: The Transformer model with BERT-fusion achieved the best overall results, reaching a WER of 0.9% on Wikipedia (compared to 14.4% for the FST).
- Robustness: In noisy, real-world conversational data (the TEL dataset), the hybrid model outperformed the standalone FST across all categories (Numbers, Units, Dates, etc.), reducing the error rate for "Misc" entities from 15.6% to 11.1%.
- Multilingual Success: The multilingual Transformer + BERT-fusion model reduced WER in Italian to 1.6% and German to 2.1%.
Limitations and Future Work
- BART Failure: Interestingly, the authors tried using BART (a popular pre-trained seq2seq model) but found it performed poorly, often inserting extra words and failing on simple entities. This suggests that general-purpose pre-training might need specific "tuning" for the constraints of ITN.
- Confidence Scoring: The hybrid model relies on a confidence score to switch between the neural model and the FST. Refining this "switch" is a key area for future improvement to ensure the system always chooses the most reliable output.
- Future Integration: The authors plan to explore deeper integration of copy attention directly with pre-trained models to get the "best of both worlds"—contextual intelligence and strict adherence to the input text.
Summary updated