Towards Language-Universal End-to-End Speech Recognition

Suyoun Kim, Michael L. SeltzerICASSP 2018 · DOI

TL;DR

This paper introduces a novel end-to-end speech recognition system that can handle multiple languages within a single model. By using a universal character set and a new language-specific gating mechanism, their approach significantly outperforms traditional multilingual methods and even individual monolingual systems, paving the way for more efficient and accessible speech technology.

What Problem Does This Paper Address?

As voice-activated assistants and speech recognition systems become more prevalent, supporting a wide range of languages is crucial. Traditionally, this involves training and maintaining separate models for each language, which is resource-intensive and particularly challenging for low-resource languages due to data scarcity. Existing multilingual approaches often involve sharing parameters across models but still struggle with efficiency and performance, especially when dealing with many languages or code-switching scenarios. This paper aims to bridge this gap by developing a single, language-universal speech recognition system that can recognize any language it's trained on without requiring language-specific components like pronunciation lexicons.

Methods

The core of this work is an end-to-end speech recognition system based on Connectionist Temporal Classification (CTC). Instead of traditional acoustic models that rely on phonemes and pronunciation dictionaries, their system directly maps acoustic features to graphemes (characters).

The key innovations are:

  • Universal Character Set: Instead of separate output layers for each language, they propose a single, unified character set that encompasses all characters from all languages in the training data. This allows a single model to predict characters across different languages.
  • Language-Specific Gating Mechanism: To address the issue that the same grapheme might have different phonetic pronunciations across languages (e.g., the letter 'i'), they introduce a novel gating mechanism. This mechanism uses the language identity to modulate the network's internal representations in a language-specific way, effectively allowing the model to adapt its internal processing based on the language being recognized. This is achieved by multiplying the hidden layer outputs with language-dependent gates.

Datasets

The experiments were conducted using data from the Microsoft Cortana personal assistant task. The datasets included:

  • English (EN), German (DE), and Spanish (ES)
  • For each language: 150 hours of training data, 10 hours of validation data, and 10 hours of test data.

No pronunciation lexicon or external language model was used in the experiments.

Key Insights and Ideas

  • Unified Model Efficiency: A single model for multiple languages drastically simplifies deployment and maintenance compared to managing numerous individual models.
  • Universal Character Set Viability: The idea of a shared character set is effective, demonstrating that a single output layer can learn to represent characters across diverse languages, with language-specific adaptations handling phonetic variations.
  • Gating for Language Nuance: The language-specific gating mechanism is crucial. It allows the model to capture subtle, language-dependent phonetic differences associated with shared graphemes, leading to significant performance improvements.
  • Initialization Power: The language-universal model, especially with gating, serves as an excellent starting point for training monolingual models, particularly when target language data is scarce. This transfer learning capability is a major benefit.
  • Code-Switching Potential: The universal character set approach shows promise for building systems that can dynamically code-switch between languages during decoding, even without explicit language identification signals.

Key Results

  • The proposed language-universal model with gating significantly outperformed individual monolingual models and conventional multilingual approaches.
  • Relative improvements in Character Error Rate (CER) compared to monolingual baselines were:
  • English (EN): 10.7%
    • German (DE): 11.4%
    • Spanish (ES): 14.1%
  • Relative improvements in Word Error Rate (WER) were:
  • English (EN): 7.0%
    • German (DE): 8.6%
    • Spanish (ES): 11.1%
  • The gated model showed further improvement when trained on three languages compared to two.
  • When used as an initialization strategy for monolingual models, the gated language-universal model outperformed other pre-training methods, including bootstrapping from a high-resource English model.
  • A bilingual (EN+ES) model without gating performed comparably to monolingual models, suggesting its potential for code-switching.

Limitations and Future Work

  • Language Identification Assumption: The current model assumes language identity is known during training and decoding.
  • Future Work:
    • Investigating how to add a new language to an already-trained multilingual system.
  • Exploring the use of external language models to further improve performance.
  • Further research into code-switching scenarios.

Summary updated

BibTeX
@inproceedings{kim2018towards,
  author = {Suyoun Kim and Michael L. Seltzer},
  title = {{Towards Language-Universal End-to-End Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8462201},
}
Figure