TL;DR
This paper describes a system for automatic simultaneous translation, tackling both text-to-text and audio-to-text translation. The authors demonstrate that their STACL-based system achieves low latency, particularly in text translation, even with limited training data, suggesting a promising direction for real-time translation applications.
What Problem Does This Paper Address?
Automatic simultaneous translation is a challenging task that aims to translate spoken or written language in real-time, mimicking human simultaneous interpreters. This involves not only translating content accurately but also doing so with minimal delay. The paper addresses the challenge of achieving low latency in both text-to-text and audio-to-text translation scenarios, especially when dealing with limited training resources. Existing systems often struggle to balance accuracy with the speed required for true simultaneous interpretation.
Methods
The system employs a two-part approach:
- Text-to-Text Translation: The core of the text translation system is the STACL (Simultaneous Translation with Implicit Anticipation and Controllable Latency) model. STACL is designed for simultaneous interpreting and utilizes a "wait-k" strategy. This means the model waits for k words of the source text before beginning to translate. During inference, the model decodes one word at a time, completing any remaining untranslated sentences once the source is fully processed. Byte Pair Encoding (BPE) is used for data preprocessing.
- Audio-to-Text Translation: For audio-to-text translation, the system uses a pipeline approach. First, the DeepSpeech2 model, an end-to-end automatic speech recognition system, is employed to transcribe segments of Chinese audio into Chinese text. This transcribed text is then fed into the STACL model to perform the translation into the target language (English in this case). This effectively breaks down the audio-to-text task into a speech recognition step followed by a text-to-text translation step.
Datasets
The following datasets were used:
- Zh-En Text Translation Dataset: Extracted from the AST dataset provided by the NAACL workshop. It contains 37,901 parallel Chinese-English sentence pairs.
- En-Es Text Translation Dataset: Obtained from the United Nations Parallel Corpus, containing 21,911,121 parallel English-Spanish sentence pairs.
- Audio-to-Test Dataset (Chinese Speech Recognition): The system utilizes a pre-trained DeepSpeech2 model trained on the AISHELL dataset, an open-source Mandarin speech corpus.
Key Insights and Ideas
- "Wait-k" Strategy for Latency Control: The STACL model's "wait-k" strategy is a key mechanism for controlling translation latency. By explicitly defining how many source words to wait for, the system can balance the need for context with the demand for speed.
- Pipeline for Audio-to-Text: Decomposing the audio-to-text problem into speech recognition followed by text translation is a practical approach. It leverages existing powerful models for each sub-task.
- Low Latency with Limited Data: A significant finding is that the STACL model can achieve low latency even with a small number of training samples, suggesting its efficiency and potential for scenarios where large datasets are unavailable.
- Trade-off in Audio-to-Text: The paper highlights that decoding only one character at a time in the audio-to-text pipeline can lead to excessive delay. The solution adopted was to decode two characters at once to mitigate this.
Key Results
- Zh-En Text Translation:
- With
wait-k = 1: Average Latency (AL) of -1.28, BLEU of 14.86.
- With
- With
wait-k = 3: AL of -0.52, BLEU of 14.84. - The results suggest that the dataset might not be sufficient for achieving higher accuracy, as accuracy remained similar across different
wait-kvalues. - En-Es Text Translation:
- AL of -1.61, BLEU of 11.82. (Trained with
max_epoch = 1).
- AL of -1.61, BLEU of 11.82. (Trained with
- Audio-to-Text Translation (Chinese to English):
- Character-level Wait (CW) value of 19.21.
- BLEU of 7.3.
- The system achieved high delay in the audio translation task, which is noted as a limitation.
Limitations and Future Work
- High Delay in Audio Translation: The authors explicitly state that the system can only generate results with a high delay in the audio translation task.
- Dataset Sufficiency: For the Zh-En translation task, the results suggest that the dataset might not be sufficient for achieving higher accuracy.
- Future Work: The authors plan to conduct further study on the speech recognition strategy to improve the audio translation performance.
Summary updated