A data-driven methodology for the production of multilingual conversational systems

Ossama Emam, Jorge Gonzalez, Carsten Günther, Eric Janke, Siegfried Kunzmann, Giulio Maltese, Claire Waast-RichardInterspeech 2000 · DOI

TL;DR

This paper presents a unified, data-driven approach to building multilingual conversational systems, focusing on a Directory Assistance service. By sharing acoustic and language models across languages and leveraging a common architecture, they demonstrate a more efficient path to multilingual system deployment with minimal changes to core components.

What Problem Does This Paper Address?

As conversational systems become more prevalent, the demand for multilingual support, especially in diverse regions like Europe, is rapidly increasing. Traditional approaches often tackled multilinguality by developing separate components for each language, which is inefficient and difficult to scale. This paper addresses the challenge of creating a truly multilingual conversational system by proposing a methodology that integrates multiple languages within a single speech-to-speech pipeline, from acoustic modeling to natural language understanding, aiming for a unified and scalable solution.

Methods

The authors propose a data-driven methodology that emphasizes sharing of resources across languages. Key aspects include:

  • Multilingual Phonology: Defining a common, simplified phoneme set that covers multiple languages, reducing the total number of phonetic units compared to individual language alphabets. This involves mapping language-specific phonetic symbols to a unified set, often using IPA and SAMPA as references.
  • Multilingual Acoustic Models: Training Hidden Markov Models (HMMs) using a common acoustic front-end (MFCCs) and a shared decision network. The process involves initial monolingual model training, followed by merging data and using forward-backward refinement with a k-means procedure for parameter estimation. They also explore Maximum a Posteriori (MAP) adaptation.
  • Multilingual Language Models: Combining trigram-based and class-based language models. A crucial technique is tagging words with language identifiers to prevent erroneous cross-language decoding, especially when merging vocabularies.
  • Multilingual Natural Language Understanding (NLU):
    • Classer: Identifies semantic concepts (e.g., "Phone Number," "Name") using the same tags across languages, requiring minimal language-specific knowledge beyond the words within the concepts.
  • Canonicalizer: Converts identified concept values into language-independent, standard representations for backend requests.
  • Parser: Generates a semantic parse tree with language-agnostic tags and labels, based on meaning and identified concepts.
  • Unified Architecture: The system employs a hub-and-spoke architecture where components like the Telephony Interface, Speech Recognition Engine, NLU Control Center, Dialog Manager, and TTS Engine communicate via an IVR HUB. This architecture is designed to be largely language-independent, facilitating the addition of new languages.

Datasets

  • SpeechDat: Used as a source for real telephony data for acoustic model training.
  • Internal Sources: Also contributed to the real telephony data for acoustic model training.
  • Downsampled 22 kHz database (8 kHz): Used for initial acoustic model training.
  • Corpora for Language Modeling: Used for training trigram and class-based language models. Specific sizes mentioned are 0.5 to 1 million words for the corpus.
  • Directory Assistance Service Data: Used for training the NLU components.

The paper mentions that the initial system was built for French, German, and British English, and Spanish was added later. The addition of further languages is underway.

Key Insights and Ideas

  • Unified Phonology is Key: A significant reduction in the number of phonetic units is achievable by defining a common phonology, which simplifies acoustic modeling for multiple languages.
  • Language Tagging for Robustness: Explicitly tagging words with their language is crucial for preventing decoding errors and ensuring correct language model usage, especially in mixed-language scenarios.
  • Language-Agnostic NLU: The NLU components (Classer, Canonicalizer, Parser) can be made largely language-independent by focusing on semantic concepts and using common tagging schemes, greatly simplifying multilingual system development.
  • Architectural Reusability: The proposed architecture allows for the addition of new languages with minimal modifications to the core dialog system components, primarily requiring new linguistic and acoustic corpora.

Key Results

  • Acoustic Model Performance: Decoding experiments using finite state grammars showed only a small incremental degradation when more languages were incorporated into the acoustic models. For example, a system trained on English, French, and German had word error rates ranging from 11.5% (monolingual English) to 23.2% (MAP approach for Gr, Fr, En).
  • Language Model Performance: For a multilingual language model trained for British English, Spanish, and German, word error rates ranged from 15% (German) to 25% (British English), with Spanish at 19.8%.
  • Phonetic Unit Reduction: The initial multilingual phonology reduced the total number of phonetic units from 354 to 300 for seven languages, and further to 129 for a set of twelve languages.

Limitations and Future Work

  • MAP Approach Performance: The paper notes that the Maximum a Posteriori (MAP) adaptation approach did not perform as well as other methods in some decoding experiments, suggesting it might be sensitive to the specific test domain or incremental training.
  • Future Work: The authors plan to explore further advancements in acoustic modeling, including Linear Discriminant Analysis, Bayesian Information Criterion for model complexity, and speaker clustering methods. They also aim to continue adding more languages to the system.

Summary updated

BibTeX
@inproceedings{emam2000data,
  author = {Ossama Emam and Jorge Gonzalez and Carsten Günther and Eric Janke and Siegfried Kunzmann and Giulio Maltese and Claire Waast-Richard},
  title = {{A data-driven methodology for the production of multilingual conversational systems}},
  booktitle = {Interspeech},
  year = {2000},
  doi = {10.21437/icslp.2000-793},
}
Figure