TL;DR
This paper explores a cost-effective method for rapidly developing speech translation (ST) systems for new language pairs, particularly under-resourced ones like English-Pashto. They demonstrate that audio recordings of human consecutive interpreters (CI) can serve as valuable training data for machine translation models, significantly improving performance even with limited parallel text.
What Problem Does This Paper Address?
Developing robust speech translation (ST) systems traditionally requires vast amounts of parallel data (source text aligned with target text). Acquiring this data is expensive and time-consuming, often limiting ST development to well-resourced language pairs. This bottleneck hinders the deployment of ST for critical communication needs in situations where new language pairs suddenly become important (e.g., international crises). The authors aim to overcome this by leveraging a readily available, albeit noisy, data source: recordings of human interpreters.
Methods
The core idea is to use "parallel speech" (pSp) audio – recordings of interpreter-mediated communication – as training data for machine translation (MT) models. This involves:
- Automatic Speech Recognition (ASR): Transcribing the source and interpreted speech from the audio recordings. This is a crucial step, especially for under-resourced languages where ASR accuracy can be a challenge.
- Translation Model (TM) Training: Using the automatically transcribed pSp audio to train statistical MT models.
- Leveraging Consecutive Interpretation (CI): The paper specifically focuses on CI, where speakers and interpreters take turns. This is contrasted with simultaneous interpretation (SI), where interpreters speak concurrently with the source. CI is hypothesized to be more "complete" and "equivalent" due to fewer time constraints, potentially making it a better source for training data.
- Data Augmentation: Investigating the impact of using pSp audio in addition to existing parallel text data.
Datasets
- English/Pashto Parallel Speech (pSp) Corpus: This is the primary dataset, consisting of audio recordings from consecutive interpretation scenarios (specifically, English-speaking soldiers interviewing Pashto-speaking Afghans, as part of the DARPA TransTac project). It includes native speech (interviewer/respondent) and interpreter speech, along with manual reference transcriptions and translations.
- English/Pashto Parallel Text Corpus: A traditional corpus of manually translated Pashto respondent utterances, used for comparison and as a baseline.
Key Insights and Ideas
- Interpreters as a Data Source: Human interpreters, whose work is already being done, can be a surprisingly effective and low-cost source for generating training data for ST systems.
- CI vs. SI for Training Data: Consecutive interpretation (CI) audio might be more beneficial for training MT models than simultaneous interpretation (SI) audio. This is attributed to CI's less severe time constraints, leading to more complete and accurate interpretations, and potentially simpler sentence alignment.
- Robustness to ASR Errors: Even with a significant Word Error Rate (WER) in the ASR system for Pashto (a known challenge for under-resourced languages), the pSp audio still provides valuable training signals.
- Complementary to Parallel Text: pSp audio can not only replace parallel text but also augment it, leading to further performance improvements.
Key Results
- pSp Audio as Replacement: TMs trained solely on automatically transcribed CI audio achieved comparable text translation performance (measured in BLEU score) to TMs trained on a smaller amount of manually translated text.
- Performance Gains with Augmentation: Adding pSp audio to a baseline system trained on parallel text resulted in statistically significant improvements in both text and speech translation performance. For example, system D+C+F (parallel text + CI audio + additional CI audio with higher WER) showed significant gains over system D (parallel text only).
- Pashto Challenges: The Pashto ASR system exhibited higher WERs, particularly on interpreter speech (up to 44.9%), compared to native speech. Despite this, the approach proved effective.
Limitations and Future Work
- ASR WER: The performance is still impacted by the Word Error Rate of the ASR system, especially for the under-resourced Pashto language.
- Word Confidence Filtering: The authors did not apply word-confidence based filtering of ASR hypotheses, suggesting this as a potential area for future improvement.
- Scaling Up: Future work should address larger amounts of pSp audio and more language pairs to further validate the hypotheses.
- CI vs. SI Further Investigation: More research is needed to definitively confirm the superiority of CI audio over SI audio for training data.
- Automatic Segmentation: While manual segmentation was used, the authors note that automatic segmentation and speaker identification could likely achieve similar performance.
Summary updated