TL;DR
This paper introduces a unified language modeling approach for multimodal voice search that simultaneously improves the accuracy of displaying business names (Inverse Text Normalization or ITN) and cleans up noisy training data. By treating both tasks as variations of the same language modeling problem, they achieve a 25% reduction in model size and a 25% improvement in accuracy, making voice search more reliable and data collection more efficient.
What Problem Does This Paper Address?
Multimodal voice search applications, especially on mobile devices, face two significant hurdles. First, when a user searches for a business, the system needs to display the exact original business name for confirmation. However, speech recognition systems often "normalize" text (e.g., converting "Rite Aid" to "Right Aid" or "4X4" to "Four by Four") to improve recognition accuracy. This necessitates an "Inverse Text Normalization" (ITN) step to convert the recognized, normalized text back to its original form. Without accurate ITN, users might see confusing or incorrect business names.
Second, the language models (LMs) that power these systems are trained on vast amounts of data, which is often "corrupted" – inconsistently labeled, incompatible, or simply inaccurate. Cleaning this data is a laborious process, hindering the development of robust LMs. This paper tackles both the ITN challenge and the data cleanup problem by proposing a unified language modeling framework.
Methods
The core innovation is a unified language modeling (LM) framework that treats both Text Normalization (TN) and Inverse Text Normalization (ITN) as language modeling tasks.
- Standard TN: The paper first explains standard TN, which involves rule-based conversions for symbols/digits, dictionary-based replacements for homonyms/abbreviations, and word breaking. This process improves speech recognition but creates the need for ITN.
- Baseline ITN: A simple unigram-based ITN approach was initially explored, but it proved insufficient, making mistakes on over 23% of business listings.
- N-Gram LM for ITN: The authors then formulated ITN as an optimization problem similar to Automatic Speech Recognition (ASR). They proposed using an N-gram language model (specifically, trigrams) to estimate the probability of a word sequence in its display form, given the spoken form. This approach significantly reduced errors to just 3%.
- Unified Implicit ITN LM: To avoid maintaining separate LMs for speech recognition and ITN, they developed a unified LM. This framework uses a common set of "word units" that represent both the display form and the spoken form (e.g.,
[Rite]for the spoken form of "Rite"). The language model captures context for disambiguation, and the grammar (specifically, a Context-Free Grammar or CFG) embeds the display form. This unified approach eliminates the need for a separate ITN decoder. - Data Cleanup as ITN: The paper argues that cleaning a corrupted corpus can be viewed as an ITN task. The same unified LM framework can be applied, where the "spoken form" is the corrupted data, and the goal is to recover the "display form" (the correct, normalized version).
Datasets
- Business Listings: The primary dataset used for ITN evaluation consists of business listings from the Seattle metropolitan area. Initially, 181,438 listings were used, with approximately 30% being modified by the TN process.
- Corrupted LM Training Corpus: A "corrupted" LM training corpus was derived from Tellme's telephony-based directory assistance application, specifically for the Seattle metropolitan area. This data was considered corrupted due to inconsistent word units and deviations from TN guidelines.
- Multimodal Local Search Log: Queries from a multimodal local search application log were used to identify business listing queries.
Key Insights and Ideas
- Unified LM is Key: The most significant insight is that ITN and data cleanup can be elegantly handled by a single, unified language modeling framework. This simplifies the system architecture and improves efficiency.
- ITN is Crucial for Multimodal: For multimodal applications where users see the recognized output, ITN is not just a nice-to-have but a necessity to ensure user trust and prevent confusion.
- Data Quality Matters Immensely: Even a "corrupted" corpus, when cleaned using this ITN-based approach, yields substantial improvements in accuracy and model size. This highlights the critical role of data quality in LM performance.
- Not All TN is Beneficial for Multimodal: Surprisingly, the authors found that removing certain TN practices (like homonym replacement, hyphens, and simple compound word breaking) did not significantly harm accuracy in multimodal settings and even reduced model size and processing time. This suggests a need for more nuanced TN strategies in multimodal contexts.
Key Results
- ITN Accuracy: The proposed unified ITN LM approach reduced mistakes on business listings from over 23% (baseline unigram) to just 3.4%.
- Model Size and Accuracy: The unified LM framework resulted in a model that is 25% smaller and 25% more accurate compared to a baseline approach.
- Data Cleanup Impact: Cleaning the "corrupted" LM training corpus led to a 12% relative gain in Top1 accuracy and an additional 25.3% gain after cleanup, while also reducing the CFG size.
- Efficiency Gains: The unified LM approach was roughly 20% larger in size and 40% more expensive in CPU time compared to a baseline CFG without ITN capability, but significantly more affordable than a two-LM approach.
- Impact of TN Practices: Removing homonym rules, hyphens, and compound word breaking from TN practices led to smaller CFG sizes and faster recognition (up to 20% faster). However, completely removing all TN rules drastically increased recognition cost (3 times more expensive).
Limitations and Future Work
- Manual Synonym List: The current process for creating synonym lists for data cleanup is largely manual, though the authors are investigating automatic approaches using web search and click-through data.
- N-best List Redundancy: The unified ITN LM can still produce redundant or similar alternatives in the N-best list (e.g., various forms of "In-N-Out"). Future work may involve post-processing to filter these or using search engines to evaluate alternatives.
- Translation Models for Training Data: The authors are exploring the use of translation models to expand the synonym training set, providing richer contexts for the ITN LM.
- Discretion in Displaying Forms: The paper suggests that preserving the display form should be done with discretion, implying that not all variations are equally important or beneficial to present to the user.
Summary updated