Unified Verbalization for Speech Recognition & Synthesis Across Languages

Sandy Ritchie, Richard Sproat, Kyle Gorman, Daan van Esch, Christian Schallhart, Nikos Bampounis, Benoît Brard, Jonas Fromseier Mortensen, Millie Holt, Eoin MahonInterspeech 2019

TL;DR

This paper introduces a unified approach to converting written text to spoken forms for both speech recognition (ASR) and text-to-speech (TTS) systems. By sharing verbalization grammars and leveraging linguistic commonalities, they significantly reduce development time and improve scalability across languages, particularly for complex number names.

What Problem Does This Paper Address?

Automatic speech recognition (ASR) and text-to-speech (TTS) systems both need to handle the conversion of written text to its spoken equivalent for various categories like numbers, dates, and currencies. This process, known as "verbalization," is often handled by separate, custom-built systems for each language and each category. This leads to significant duplication of effort, maintenance overhead, and challenges in scaling to a large number of languages. Specifically, number names present a particular challenge due to their diverse linguistic structures (e.g., different bases, ordering of factors, and "flop" phenomena where the order of addends or multiplicands is reversed). The paper aims to create a more efficient and scalable solution by unifying these verbalization processes.

Methods

The core of the paper's contribution lies in two main areas:

  • Improved Number Names Induction: They enhance a previous system for inducing number name grammars from a small set of labeled examples. The improvements focus on better handling of "addend flops" (e.g., "four-and-twenty" vs. "twenty-four") and "multiplicand flops" (e.g., "two hundred" vs. "hundred two"). The new induction system uses real arithmetic to find the optimal parse tree for number factors, making it more robust to typological variations.
  • Unified Grammar Template Framework: They propose a system of customizable, cross-lingual, finite-state template grammars. This framework allows for shared verbalization logic across ASR and TTS systems. The system uses a base class for common semiotic categories, with ASR and TTS-specific subclasses to handle their distinct requirements (ASR accepting multiple variants, TTS emitting a single form). This approach leverages linguistic commonalities and markedness principles to create default verbalizations, with parameters to handle language-specific variations like word order, allomorphy, and morphological concord.

Datasets

The paper mentions using training data consisting of digits mapped to their verbalizations (e.g., 123 → one hundred twenty three). They also refer to a questionnaire-based approach to gather data for new languages from language consultants. A dataset of verbalizations for numbers 1-100 and powers of ten is available at https://github.com/google/UniNum.

Key Insights and Ideas

  • Linguistic Commonalities as a Foundation: The paper highlights that while number names are highly diverse, other semiotic classes (like currencies) share significant structural similarities across languages. This insight is crucial for building a unified, template-based system.
  • Markedness as a Design Principle: By identifying typologically marked features (like reversed word order in currency expressions), the system can default to the unmarked form and use specific flags or parameters to accommodate marked variations, simplifying grammar development.
  • Unified ASR/TTS Logic: The realization that ASR and TTS verbalization logic can be largely shared, with only minor differences in output handling (multiple variants vs. single output), is a key enabler for code reduction and efficiency.
  • Parameterization for Scalability: The template-based approach, where language-specific variations are handled by flipping parameters rather than writing entirely new grammars, is the core mechanism for achieving scalability.

Key Results

  • Significant Code Reduction: For the money class, the new template system reduced custom handwritten code from approximately 200 lines to just 25 lines for specifying constants and parameters.
  • Improved Handling of Number Name Phenomena: The enhanced induction system better supports complex number naming structures like addend and multiplicand flops, which were problematic for previous systems.
  • Faster Development and Scalability: The unified approach and template-based system significantly reduce development cost and time, paving the way for faster scaling to new languages.
  • Support for Linguistic Features: The system effectively handles phenomena like allomorphy, morphological concord, and marked word orders without requiring complex custom architectures for each language.

Limitations and Future Work

  • Unbounded Addend Flops: The current system does not handle unbounded addend flops (where the order of factors is reversed indefinitely), though they suggest pre- or post-processing as a future solution.
  • Tooling for Questionnaire Data: Developing tools to automatically parse questionnaire data from language consultants would further accelerate grammar generation.
  • Leveraging External Data: Utilizing resources like the Unicode Common Locale Data Repository (CLDR) could help automate the acquisition of template parameters.
  • Expanding Semiotic Classes: The framework could be extended to cover more semiotic classes beyond those discussed.
  • Increased Robustness: Further improvements are needed to enhance robustness to typological diversity, particularly for complex number name structures.

Summary updated

BibTeX
@inproceedings{ritchie2019unified,
  author = {Sandy Ritchie and Richard Sproat and Kyle Gorman and Daan van Esch and Christian Schallhart and Nikos Bampounis and Benoît Brard and Jonas Fromseier Mortensen and Millie Holt and Eoin Mahon},
  title = {{Unified Verbalization for Speech Recognition \& Synthesis Across Languages}},
  booktitle = {Interspeech},
  year = {2019},
}

Loading related content...

Figure