Parallel combination of multilingual speech streams for improved ASR

João Miranda, João Paulo Neto, Alan W. BlackInterspeech 2012 · DOI

Abstract

This paper introduces a novel framework for enhancing Automatic Speech Recognition (ASR) by exploiting the multilingual redundancy inherent in simultaneous interpretations. In settings such as the European Parliament, a single speech is often interpreted into dozens of languages. By intersecting ASR word lattices from these different languages with Statistical Machine Translation (SMT) phrase tables, the proposed system treats interpreted speech as a cross-lingual sanity check. Experimental results demonstrate that this parallel combination approach achieves a 20.8% relative improvement in Word Error Rate (WER) for English speeches when combined with Portuguese and Spanish interpretations.

Introduction

In international organizations like the United Nations or the European Parliament, transcribing original speeches can be challenging due to non-native accents, disfluencies, and technical jargon. However, these environments provide a unique resource: simultaneous interpretations. Traditionally, ASR and Machine Translation (MT) are used sequentially (transcribe then translate). This research explores a "parallel combination" architecture that uses translated streams to assist the original transcription process, overcoming the hurdles of asynchronous, error-prone audio streams without requiring perfect manual alignments.

Methodology

The proposed system architecture leverages the flexibility of word lattices to maintain a massive space of alternative hypotheses, allowing the system to "rescue" correct words that might not be the ASR's top choice if they match a phrase in the interpreted stream.

The parallel combination architecture for multilingual ASR, showing the flow from multi-stream ASR to lattice rescoring.

The Parallel Combination Pipeline

  1. Multi-stream ASR: The system runs ASR on the original language and all available interpreted versions, producing word lattices (graphs of possible word sequences).
  2. Phrase Table Intersection: SMT phrase tables serve as a bridge. The algorithm searches for phrase pairs where the source phrase appears in the original lattice and the target phrase appears in the interpreted lattice.
  3. Temporal Alignment: To handle "interpreter lag," a maximum delay parameter ($\delta = 10$ seconds) ensures matched phrases refer to the same moment in time.
  4. Feature-Based Selection: A linear model scores phrase pairs based on posterior probabilities, language model scores, and "agreement" (the number of languages suggesting the same translation).
  5. Lattice Rescoring: An A* search is performed on the original lattices, providing a bonus score to paths containing high-confidence phrase pairs identified in the previous steps.

Experimental Setup

  • Training Data: SMT phrase tables and 4-gram language models were trained using the Europarl Parallel Corpus.
  • Evaluation Data: A custom set of English speeches from four European Parliament committees (Environment, Development, Internal Market, and Legal Affairs) was collected and transcribed.
  • Acoustic Models: The system utilized the Audimus hybrid ANN-MLP WFST-based recognizer.

Results and Discussion

The experiments focused on improving English ASR by combining it with Portuguese (PT) and Spanish (ES) interpretations. The results indicate that the gains from adding languages are nearly cumulative.

Word Error Rate (WER) results across different speech categories and language combinations.

As shown in the table above, the baseline English WER of 28.50% was significantly reduced. Combining English with Portuguese alone yielded a 10.0% relative improvement (25.65% WER), while the full combination (EN+PT+ES) reached 23.77% (a 16.6% relative improvement). After a second iteration involving unsupervised speaker adaptation, the WER dropped to 22.56%, representing a total 20.8% relative improvement over the baseline.

Key Insights

  • Implicit Alignment: Unlike previous methods, this approach does not require pre-aligned audio segments; alignment occurs implicitly during the phrase-table-to-lattice intersection.
  • The Power of Agreement: One of the most effective features was the "number of languages." If multiple interpretations suggest a specific translation for an English phrase, the confidence in that phrase increases significantly.

Conclusion and Future Work

This research demonstrates that multilingual redundancy is a powerful tool for improving ASR robustness. While the method currently relies on high-quality SMT phrase tables and involves significant computational complexity, future work could scale this to more languages. Additionally, the authors suggest that multiple streams could be used to specifically detect and recover Out-of-Vocabulary (OOV) words or technical terms that are often lost in standard ASR systems.

Summary updated

BibTeX
@inproceedings{miranda2012parallel,
  author = {João Miranda and João Paulo Neto and Alan W. Black},
  title = {{Parallel combination of multilingual speech streams for improved ASR}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-303},
}
Figure