Abstract: Mattia A. Di Gangi, Marco Gaido, Matteo Negri, Marco Turchi. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track).
2020.
Introduction
Direct Speech Translation (ST) aims to translate audio from a source language directly into text in a target language using a single end-to-end model. A persistent question in this field is how to best segment the target text. While Automatic Speech Recognition (ASR) traditionally favors character-level segmentation, Neural Machine Translation (NMT) has standardized on subword-level units like Byte Pair Encoding (BPE). This paper provides a rigorous comparison to determine which approach is superior for direct ST.
Methodology
The researchers utilized the S-Transformer architecture, which adapts the standard Transformer for speech by incorporating 2D convolutional layers to process audio spectrograms. They compared two primary segmentation strategies:
- Character-level: Predicting individual letters and punctuation.
- Subword-level (BPE): Predicting tokens from a vocabulary of 8,000 subword units.
The study also explored multilingual settings, testing "concat" and "sum" methods for language identification in models translating English into multiple target languages.
Datasets
To ensure robust findings, experiments were conducted on three major benchmarks:
- Augmented Librispeech: English-to-French (200 hours).
- MuST-C: The largest ST corpus, covering English to eight target languages.
- How2: English-to-Portuguese instructional video data (300 hours).
Key Results
The benchmarking revealed a clear winner: subword-level segmentation (BPE) consistently outperformed character-level models across every language and dataset tested. On the MuST-C benchmark, BPE models provided gains ranging from 1.2 to 3.3 BLEU points.
In English-to-French translation, for instance, the BPE model achieved 30.3 BLEU compared to 26.5 for the character-level model. Similar improvements were observed on Librispeech, where the BPE model reached 17.1 BLEU, setting a strong baseline against previous literature.
Analysis and Insights
The researchers identified several reasons for the superiority of BPE:
Handling Sequence Length
Character-level models struggle as output sequences grow longer. Because BPE represents text in fewer tokens, the model can maintain better focus over the sequence. As shown in the analysis of sequence length, the performance gap (DELTA) between BPE and character-level models widens significantly as the number of tokens increases.
Memorization vs. Generalization
Character-level models often exhibit "peakiness" in their probability distributions, suggesting they rely on memorizing specific sequences. In contrast, BPE embeddings capture richer semantic and syntactic information (like common prefixes), aiding generalization. Using the Translation Error Rate (TER) metric, BPE models were found to produce better translations in 44.6% of cases, while character models only "won" in 30.2% of instances.
Conclusion
This study demonstrates that for direct speech translation, subword-level segmentation via BPE offers significant advantages in both accuracy and efficiency over character-level approaches. While massive models might utilize characters differently, BPE remains the optimal choice for standard practical applications in the field.
Summary updated