TL;DR
This paper tackles the challenge of speech translation systems struggling with errors from Automatic Speech Recognition (ASR) and punctuation. They introduce an "inverted projection" method to align system-generated transcriptions with human translations, creating training data that mimics real-world speech input. This approach significantly improves translation quality by making models more robust to these common errors.
What Problem Does This Paper Address?
Traditional machine translation (MT) systems excel when trained on clean, well-structured text. However, when translating spoken language, a cascade of systems is typically used: speech is first transcribed by an ASR system, then potentially segmented by an automatic punctuation system, and finally translated by an MT system. This pipeline amplifies errors. Spoken language itself can be less structured than written text (e.g., disfluencies, partial sentences). Crucially, ASR systems are imperfect, introducing word-level transcription errors, and automatic punctuation can create unnatural sentence boundaries. These errors, especially when combined, lead to a significant "train-test mismatch" where models trained on clean data perform poorly on real-world, error-prone speech inputs. This paper aims to bridge this gap by making MT systems more robust to these ASR and punctuation errors.
Methods
The core innovation is the "inverted projection" technique. This method addresses the train-test mismatch by creating training data that reflects the errors encountered during real-world speech translation.
Here's how it works:
- Levenshtein Alignment: The paper first establishes an alignment between the ASR system's transcription of the audio and the human-written transcript of the same audio. This alignment helps map words and segments between the two.
- Projection:
- Token Robustness: In a simpler approach (similar to prior work), human sentence boundaries are projected onto the ASR output. This exposes the MT model to ASR token errors but still uses clean human segment boundaries, leading to a train-test mismatch.
- Segment Robustness: This is where the "inverted projection" comes in. Instead of projecting human boundaries onto system output, they project the system's segment boundaries (derived from automatic punctuation) onto the human transcript. To align the target (gold translation) with these new system-derived segments, they use a length-ratio heuristic based on token counts. This creates training pairs where the source side has system-generated segment boundaries.
- System Robustness: This approach combines the benefits of Segment Robustness by using system segment boundaries and also injects ASR token errors into the source side. The gold translations are then re-segmented to match the projected system segments.
Essentially, the goal is to train the MT model on source sentences that mimic the output of the ASR and punctuation pipeline, thus bridging the train-test mismatch.
Datasets
- IWSLT EnDe (English to German) speech translation scenario:
- Training Data: A mix of WMT and human-transcribed IWSLT data (2400 talks, 0.25M sentence pairs), plus 4.6M sentence pairs from WMT 2018 EnDe corpus.
- Dev Set: IWSLT tst2014 (14 TED talks, ~1,200 sentences).
- Test Set: IWSLT tst2015 (12 TED talks, ~1,200 sentences).
- ASR Output: Obtained from Google's Speech-to-Text API.
- YouTube Dataset:
- Test Set: 88 English videos translated into German, consisting of 920 sentences. This dataset was used to evaluate the generalization of models trained on IWSLT.
Key Insights and Ideas
- Segmentation Errors Matter More: The paper's most striking finding is that making MT models robust to segmentation errors (i.e., incorrect sentence boundaries) is more crucial for speech translation than solely addressing token-level transcription errors, especially when ASR quality is already high.
- Bridging the Train-Test Mismatch is Key: The core problem isn't just ASR errors, but the mismatch between how training data is prepared (clean) and how the system operates at inference time (with ASR/punctuation errors). The inverted projection directly tackles this mismatch.
- System Segments are Necessary: While using human segments is standard, the paper demonstrates that training with system-generated segments is essential for achieving robust performance on real-world speech inputs.
- High WER Filtering Helps: Removing ASR sentences with very high Word Error Rates (WER ≥ 50%) during fine-tuning further boosts performance, suggesting that extremely garbled inputs can be detrimental even to robust models.
Key Results
- IWSLT Results:
- The System Robustness approach (using system segments and ASR errors) achieved significant gains.
- The best System Robustness model, after filtering out high-WER ASR sentences, yielded a gain of over 1 BLEU point compared to the baseline.
- The Token Robustness approach (using ASR tokens but human segments) showed no improvement, highlighting the dominance of segmentation errors.
- YouTube Results:
- Models fine-tuned on IWSLT data generalized well to the YouTube dataset.
- The System Robustness approach again showed an improvement of about 1.2 BLEU points over the baseline on this new dataset.
Limitations and Future Work
- ASR Quality: The paper notes that their English ASR system has a relatively low WER (5.5% on tst2015). They hypothesize that they could have replicated prior work showing improvements from ASR token errors if they had used a weaker ASR model.
- Domain Mismatch: For the YouTube results, there's a domain mismatch between the IWSLT training data and the YouTube test data. They expect even higher scores if models were trained directly on YouTube data.
- Future Work:
- Investigate languages with higher ASR WER.
- Explore incorporating MT targets during training to address data bottlenecks.
- Investigate document-level speech translation to potentially eliminate segmentation altogether.
Summary updated