A Mostly Data-Driven Approach to Inverse Text Normalization

Ernest Pusateri, Bharat Ram Ambati, Elizabeth Brooks, Ondrej Platek, Donald McAllaster, Venki NageshaInterspeech 2017

TL;DR

This paper presents a novel "mostly data-driven" approach to Inverse Text Normalization (ITN), the process of converting spoken-form words from speech recognition into their written equivalents. By framing ITN as a labeling problem and employing a compact bi-directional LSTM, the authors achieve performance comparable to traditional rule-based systems with significantly less manual effort and more flexibility.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often output sequences of spoken-form tokens (e.g., "one forty one" instead of "141"). To make this output readable and usable, it needs to be converted into its written form – a process called Inverse Text Normalization (ITN). This transformation is particularly challenging for entities like numbers, dates, times, and addresses, which have complex spoken-to-written mappings. Existing ITN systems often rely heavily on handcrafted rules and grammars, which are time-consuming to develop, maintain, and adapt to different languages or domains. This paper aims to bridge this gap by proposing a more data-driven approach that leverages machine learning to handle these transformations, reducing the reliance on extensive manual rule creation.

Methods

The core of the proposed approach is to cast ITN as a token labeling problem. For each spoken-form token, the system assigns a "label" that specifies a series of edits (rewrite, prepend, append, space insertion) needed to produce the corresponding written-form segment. This labeling is then performed by a compact bi-directional Long Short-Term Memory (LSTM) neural network.

The process involves:

  1. Label Definition: A structured label format is defined with six fields: Rewrite, Prepend, Append, Space, Post Start, and Post End. Each field can be populated with specific actions or grammars. Finite-state transducers (FSTs) are constructed for each possible action within these fields.
  2. Label Inference: To train the LSTM, spoken-form/written-form pairs are converted into spoken-form/label-sequence pairs using an FST-based pipeline. This pipeline involves expanding tokens, applying label actions, rendering the output, and optionally applying post-processing grammars.
  3. Bi-directional LSTM: The trained LSTM takes token embeddings and handcrafted features (like capitalization and rewrite table compatibility) as input and outputs a probability distribution over the defined labels for each token.
  4. Post-processing: For specific complex transformations (e.g., currency, relative time, roman numerals), post-processing grammars are applied to regions tagged by the Post Start and Post End label fields.

This approach is "mostly data-driven" because the core transformation is learned by the LSTM from data, with only a few simple rules and grammars needed for specific cases or for defining the label actions.

Datasets

The paper uses data from an English virtual assistant application. They generate training and test data by using an existing rule-based system to convert spoken-form data to written-form. They estimate the sentence accuracy (SA) of this rule-based system to be around 99%. For evaluation, they use a large set of 800,000 randomly selected utterances, as well as targeted test sets for specific entity types (TIME, MATH, ADDRESS, MEASURE, DATE, CURRENCY).

Key Insights and Ideas

  • Framing ITN as a Labeling Problem: This is a crucial insight. By breaking down the complex ITN task into assigning labels to individual tokens, the problem becomes amenable to sequence labeling models like LSTMs.
  • Compact Bi-directional LSTM: The choice of a compact bi-directional LSTM is significant. It offers a good balance between performance and computational efficiency, making it faster to train and run inference on compared to more complex sequence-to-sequence models.
  • Leveraging Features: The inclusion of simple features like capitalization and features derived from rewrite tables proves effective. This demonstrates that even basic linguistic cues can significantly aid the model in making correct decisions.
  • Data-Driven Flexibility: The ability to change system behavior by modifying training data, rather than rewriting complex rules, offers a significant advantage in terms of development and maintenance.

Key Results

  • The proposed "mostly data-driven" approach achieves high sentence fidelity (SF), approaching the performance of a highly accurate rule-based system.
  • With 1 million training utterances, the system achieves an overall SF of 99.62%, very close to the rule-based system's estimated 99% SA.
  • The capitalization feature alone provides significant gains, improving SF from 99.26% to 99.62% with 1M training utterances.
  • Features derived from the rewrite table provide further, albeit smaller, improvements, particularly for handling unseen measurement units.
  • The system performs well on targeted test sets, with SF exceeding 90% for most categories when trained with 1M utterances, demonstrating its ability to handle complex transformations.

Limitations and Future Work

  • Data Dependency: While data-driven, the system's performance is still dependent on the quality and quantity of training data. The authors note that for infrequent or out-of-vocabulary units, the system might not generalize as well without specific training examples.
  • Targeted Data Selection: The authors found it beneficial to over-represent utterances with significant transformations in the training data. This suggests that more sophisticated data selection strategies could further improve performance.
  • Error Correction: The paper mentions that they manually corrected some likely errors in the training data to improve performance on specific types of utterances (e.g., time expressions). This highlights the ongoing need for data curation and error analysis.
  • Future Work: The authors suggest that more general attention-based sequence-to-sequence models could be explored, although they opted for a more compact LSTM for efficiency. Further improvements could come from more advanced data selection techniques and potentially incorporating more linguistic knowledge into the feature set.

Summary updated

BibTeX
@inproceedings{pusateri2017mostly,
  author = {Ernest Pusateri and Bharat Ram Ambati and Elizabeth Brooks and Ondrej Platek and Donald McAllaster and Venki Nagesha},
  title = {{A Mostly Data-Driven Approach to Inverse Text Normalization}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure