Abstract
Machine translation (MT) systems frequently produce disfluent or "clunky" text that standard speech synthesizers struggle to process naturally. This study investigates three strategies to improve the intelligibility of spoken MT output: strategic pause insertion, the use of fillers for untranslated words, and phonetic optimization via N-best list selection. Experimental results demonstrate significant reductions in Word Error Rate (WER) across these methods, suggesting that "gracefully handling" disfluencies is more effective than ignoring them in speech-to-speech translation pipelines.
Introduction
In a standard speech-to-speech translation (S2ST) pipeline, the transition between Machine Translation (MT) and Text-to-Speech (TTS) synthesis is often a bottleneck. While TTS engines are optimized for fluent, natural language, MT output is frequently marred by errors such as untranslated foreign words, awkward reordering, and poor lexical choices. When a synthesizer attempts to read a disfluent sentence like "A bird sat on of the tree" as perfect English, the resulting audio can be significantly harder to understand than the source text. This research aims to bridge this gap by enabling synthesizers to adapt to the inherent imperfections of MT output.
Methodology
The researchers proposed and evaluated three distinct strategies to enhance the intelligibility of synthesized MT output.
1. Strategic Pause Insertion
Humans naturally introduce pauses when encountering complex or erroneous structures. The study compared a default TTS pause model against an "oracle" model where human annotators marked appropriate pause locations in disfluent MT text. This approach tests whether mimicking human reading patterns for broken text improves listener comprehension.
2. Handling Untranslated Words with Fillers
When an MT system fails to translate a word (e.g., leaving a Portuguese word in an English sentence), standard TTS often produces phonetic gibberish. The authors proposed replacing these out-of-vocabulary (OOV) words with a filler ("Umm"), accompanied by short pauses and a 20% pitch reduction. This serves as a cognitive signal to the listener that information is missing, allowing them to focus on the surrounding context.
3. Phonetic Optimization via N-Best Lists
Unit-selection synthesizers sometimes produce "bad joins" or glitches when concatenating certain word sequences. By leveraging the MT system's N-best list, the researchers selected alternative translations that maintained semantic similarity (measured by METEOR scores) but offered lower unit-selection join costs, resulting in clearer audio.
Experimental Setup
The strategies were evaluated using two primary language pairs:
- Chinese–English: A phrase-based MT system (Moses) trained on the 11-million-sentence Gigaword corpus was used for pause-insertion experiments.
- Portuguese–English: A system trained on the Europarl corpus was used for the filler experiments due to its high frequency of OOV words.
Intelligibility was measured through listening tasks where subjects transcribed synthesized audio to calculate the Word Error Rate (WER).
Results and Discussion
The experimental results indicate that targeted handling of disfluencies significantly improves listener performance.
Intelligibility Gains from Fillers
The filler strategy proved highly effective at mitigating the confusion caused by untranslated words. By signaling a disfluency rather than attempting a verbatim (and likely incorrect) pronunciation, the system helped listeners maintain their grasp on the sentence structure.
As shown in the table above, the "System-Filler" approach reduced the WER in the vicinity of the error from 30.1% to 24.0%.
Phonetic Optimization Results
Selecting translations based on their "synthesizability" also yielded measurable improvements. By avoiding difficult phonetic joins, the system produced smoother audio.
The use of the "System-Alternative" strategy resulted in a drop in WER from 28.9% to 24.7% compared to the standard top-best translation. Additionally, the pause-insertion experiment showed that human-annotated pauses led to a 10% relative reduction in WER compared to the default model.
Conclusion
This research demonstrates that speech synthesis for machine translation can be significantly improved by acknowledging and signaling disfluencies rather than ignoring them. Future work will focus on automating the prediction of "disfluency-aware" pauses and training synthesizers specifically on human readings of MT output to adopt a more suitable delivery style for imperfect translations.
Summary updated