TL;DR
This paper introduces the Cross Attention Augmented Transducer (CAAT), a novel architecture for simultaneous speech and text translation that improves upon existing methods by jointly optimizing translation quality and latency. Their CAAT-based systems achieved significant BLEU score improvements over previous state-of-the-art systems at IWSLT 2021, demonstrating its effectiveness in both speech-to-text and text-to-text translation tasks.
What Problem Does This Paper Address?
Simultaneous translation, where translation begins before the source utterance is fully received, is challenging because it requires balancing translation accuracy with minimal delay. Existing approaches often fall into two categories: fixed policies (like "wait-k," which can lead to excessive latency) or flexible policies (which are harder to optimize). The authors aim to address this by developing a more effective and optimizable approach for simultaneous translation, particularly for tasks involving speech, where data scarcity can be a bottleneck. They also tackle the issue of integrating policy learning directly into the translation model for better performance.
Methods
The core contribution is the Cross Attention Augmented Transducer (CAAT) architecture. It extends the RNN-Transducer (RNN-T) model, which is typically used for speech recognition and has a monotonic constraint. CAAT removes this constraint, allowing for more flexible reordering, and augments the standard Attention Encoder-Decoder framework.
Key methodological innovations include:
- CAAT Architecture: This model separates the source attention mechanism from the target history representation, allowing for a deeper coupling of source context and target history. This is crucial for tasks like machine translation where reordering is common.
- Latency Loss Function: A novel latency loss function is introduced, designed to be "mergeable" and account for the expectation of latency across all possible expanded translation paths. This allows for fine-grained control over the latency-quality trade-off.
- Streaming Encoder: For speech processing, they employ a streaming encoder that uses block processing with both local and infinite left context, enabling efficient processing of speech data while maintaining context.
- Data Augmentation: To combat data scarcity in speech translation, they leverage techniques like self-training (generating target translations from ASR outputs) and speech synthesis (using TTS to create synthetic audio data).
Datasets
The paper utilizes several datasets for training and evaluation:
- Speech Translation Corpora: MuST-C, Europarl, and CoVoST2 (speech, transcription, and translation).
- Speech Recognition Corpora: LibriSpeech and TED-LIUM3 (speech and transcription).
- Text Translation Corpora:
- EN→DE: MuST-C, Europarl, Rapid-2019, WIT3-TED, Commoncrawl, WikiMatrix, Wikititles, and Paracrawl.
- EN→JA: WIT3-TED, JESC, kftt, WikiMatrix, Wikititles, and Paracrawl.
- Monolingual Corpora: Europarl-v10, News-crawl-2019, and News-crawl-2020 for English, German, and Japanese.
They also mention filtering and selecting subsets of these datasets, particularly Paracrawl for the EN→DE task, due to its large size.
Key Insights and Ideas
- Joint Optimization is Key: The paper strongly suggests that jointly optimizing translation quality and latency within a single model (like CAAT) is more effective than using separate, fixed policies.
- CAAT's Flexibility: By removing the monotonic constraint of RNN-T and augmenting attention, CAAT proves adept at handling the reordering inherent in machine translation, outperforming wait-k baselines.
- Data Augmentation is Crucial for Speech: For speech translation tasks, where parallel data is often limited, data augmentation techniques like self-training and TTS synthesis provide substantial performance gains.
- Cascaded Systems Offer Trade-offs: While end-to-end systems are promising, cascaded systems (ASR + MT) can still offer advantages in medium to high latency regions, though they inherently introduce more delay.
Key Results
- Significant BLEU Improvements:
- S2T: The proposed CAAT-based S2T system improved by an average of 11.3 BLEU points compared to last year's optimal systems.
- T2T: The CAAT-based T2T system improved by an average of 4.6 BLEU points compared to last year's optimal systems.
- Latency-Quality Trade-offs: CAAT consistently outperforms the wait-k baseline across various latency regimes, especially in low-latency scenarios.
- Data Augmentation Impact: Adding data through self-training and speech synthesis boosted offline speech translation performance by nearly 7 and 1.5 BLEU points, respectively, on the MuST-C(v2) dataset.
- Ensembling Benefits: Model ensembling further improved BLEU scores by roughly 0.5-1.5 points at different latency regimes for T2T translation.
Limitations and Future Work
- Computational Cost: The CAAT model's joiner module has a higher computational complexity than conventional Transformers, which they mitigate by reducing the joiner's hidden dimension and using a larger decision step size.
- Unsegmented Data: The paper notes that processing unsegmented data (like continuous speech streams without clear utterance boundaries) remains a challenge, and current methods might not be ideal. They suggest that more appropriate latency criteria might be needed for such data in the future.
- Cascaded System Latency: While cascaded systems show promise, they inherently have higher latency than end-to-end systems due to the sequential nature of ASR followed by MT. Further work is needed to reduce this gap.
- EN-JA Training Instability: The authors mention that the EN-JA translation task experienced some training instability, leading to a reduced joiner block parameter size and a more noticeable gap between CAAT and offline performance in that specific track.
Summary updated