Adapting machine translation models toward misrecognized speech with text-to-speech pronunciation rules and acoustic confusability

Nicholas Ruiz, Qin Gao, William Lewis, Marcello FedericoInterspeech 2015

TL;DR

This paper tackles the challenge of adapting machine translation (MT) models for spoken language translation (SLT) when automatic speech recognition (ASR) errors introduce noise. They propose a novel "damaging channel" that simulates ASR errors by converting clean text into phoneme sequences and then back into words, incorporating acoustic confusability. This allows for the generation of synthetic ASR-like data to train more robust MT models, leading to improved SLT performance.

What Problem Does This Paper Address?

Spoken language translation (SLT) systems typically follow a pipeline: first, an Automatic Speech Recognition (ASR) system transcribes spoken audio into text, and then a Machine Translation (MT) system translates this text. A major hurdle in this cascaded approach is the mismatch between the data used to train MT systems (clean, written text) and the output of ASR systems, which inevitably contains errors. These errors can include misrecognized words, out-of-vocabulary (OOV) terms, and incorrect punctuation.

While training MT models on actual transcribed and translated speech corpora would be ideal, such data is scarce and expensive to create. Existing methods often rely on limited amounts of transcribed speech for adaptation. This paper addresses this data scarcity by proposing a method to artificially create ASR-like errors in clean text, thereby generating synthetic data that can be used to adapt MT models to be more resilient to ASR noise.

Methods

The core of the proposed solution is a "damaging channel" that simulates ASR errors. This process involves two main stages:

  1. Learning to Adapt Data (Damaging Channel Setup):
    • Phoneme Conversion: Words from an ASR system's pronunciation dictionary (PD) are converted into phoneme sequences using a Text-to-Speech (TTS) analyzer. This TTS analyzer is preferred over the ASR's PD because it can handle OOV words, acronyms, and context-dependent pronunciations more effectively.
  2. Phoneme-to-Word SMT: A phrase-based SMT system is trained to translate these phoneme sequences back into words. This system is tuned using Minimum Error Rate Training (MERT) on a small set of actual ASR outputs and their corresponding single-best hypotheses.
  3. Acoustic Confusability Modeling (Phoneme-Level Confusion): To simulate phoneme errors that occur during ASR decoding (e.g., missing, distorted, or substituted phonemes), a phoneme-to-phoneme SMT system is trained. This system learns to generate distorted phoneme sequences based on observed ASR behavior. These distorted sequences are then used to expand the dictionary with multiple pronunciation alternatives for each word.
  4. Adapting Data (Generating Synthetic ASR Outputs):
    • Text Transformation: Clean source language texts from existing bitexts are first converted into phoneme sequences using the TTS analyzer.
  5. Simulated ASR Output Generation: These phoneme sequences are then fed through the trained phoneme-to-word SMT system (incorporating the acoustic confusability modeling) to generate synthetic ASR outputs.
  6. MT Adaptation: These synthetic ASR outputs, along with their original translations, are used as adaptation data for training the MT component of the SLT system. The authors duplicate the original bitexts so that MT can be trained simultaneously on clean and synthetic ASR data.

Datasets

  • Training and Evaluation: The experiments are conducted on the English-French TED speech translation task from the IWSLT 2014 evaluation campaign.
  • Training Data: TED talk videos released before December 31, 2010 (820 talks, 144 hours of speech) were used for training the ASR acoustic model. The MT system was trained on the TED talk training set permitted in IWSLT 2014.
  • Adaptation Data: Clean TED bitexts were used to generate synthetic ASR outputs via the proposed damaging channel.
  • Evaluation Data: The tst2012 dataset was used for evaluating the end-to-end SLT system.

Key Insights and Ideas

  • TTS as a Better Source for Phoneme Conversion: Using a TTS analyzer's text analysis module proved more effective than relying solely on an ASR system's pronunciation dictionary, especially for handling OOV words, acronyms, and numeric sequences.
  • Phoneme-Level Confusion is Crucial: Simply converting text to phonemes and back doesn't fully capture ASR errors. Modeling acoustic confusability at the phoneme level (e.g., using a phoneme-to-phoneme SMT) significantly improves the realism of the simulated ASR outputs and leads to better adaptation.
  • Damaging Channel Improves Robustness: MT models trained with synthetic ASR-corrupted data are demonstrably more robust to actual ASR errors, leading to better translation quality in the SLT pipeline.
  • Combining Lexical and TTS Pronunciations is Beneficial: A union of ASR PD pronunciations and TTS-generated pronunciations yielded better results than using either source alone, suggesting complementary strengths.
  • The "Damaging Channel" is a Cost-Effective Alternative: This method provides a way to generate large amounts of adaptation data without the prohibitive cost of recording and transcribing new speech corpora.

Key Results

  • Significant BLEU Improvements: The proposed method, particularly when using the CONCAT strategy (combining damaged and clean data), showed statistically significant improvements in BLEU scores, ranging from 0.6-0.8 over the baseline. The FILL-UP strategy also showed improvements, though weaker.
  • Phoneme Confusion Yields ~10% TER Improvement: Adding phoneme confusions to the damaging channel resulted in nearly a 10% absolute improvement in TER (Translation Error Rate) against both ASR outputs and original transcripts on the dev2010 set.
  • TTS-Driven Approach Handles OOV: The TTS-driven damaging channel successfully converted OOV words, acronyms, and numeric sequences into phoneme sequences that could be reconstructed into lexical forms, which the ASR PD-driven approach struggled with.
  • Qualitative Improvements: Examples showed that the adapted systems were better at handling punctuation errors and misrecognized contractions, leading to more coherent translations.

Limitations and Future Work

  • Forward Probability Dilution: The phoneme-to-word SMT system suffers from probability dilution when a word has many pronunciations, especially when phoneme confusions are introduced. This can lead to favoring incorrect word sequences.
  • Future Work: The authors suggest weighting probability distributions by corpus frequencies or pruning infrequent words. They also propose that TTS analyzers should generate more word pronunciations based on context to better capture real-world variations.
  • Detrimental Single TTS Pronunciation: Using only a single TTS pronunciation for each word was found to be detrimental, especially for gerunds, leading to nonsensical constructions.
  • Future Work: This highlights the need for TTS systems to generate more diverse pronunciations based on context.
  • Domain Adaptation Challenges: While improvements were seen, the FILL-UP strategy's weaker performance suggests that simply appending new phrases might not be as effective as integrating the damaged data more holistically. The authors note that concatenating corpora can lead to out-of-domain data dominating the translation model.
  • Future Work: Further research into optimal strategies for integrating synthetic data into MT training is warranted.

Summary updated

BibTeX
@inproceedings{ruiz2015adapting,
  author = {Nicholas Ruiz and Qin Gao and William Lewis and Marcello Federico},
  title = {{Adapting machine translation models toward misrecognized speech with text-to-speech pronunciation rules and acoustic confusability}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure