On Target Segmentation for Direct Speech Translation

Mattia Antonino Di Gangi, Marco Gaido, Matteo Negri, Marco Turchi

In the field of Neural Machine Translation (NMT), subword segmentation—specifically Byte Pair Encoding (BPE)—is the standard for representing text. However, in the emerging domain of Direct Speech Translation (ST), which translates audio directly to target text without intermediate transcription, the optimal output representation has remained a subject of debate. While Automatic Speech Recognition (ASR) often favors character-level outputs to handle open vocabularies, NMT favors subwords for efficiency and semantic grouping.

This research from AppTek and Fondazione Bruno Kessler provides a rigorous comparison between character-level and subword-level target segmentation, concluding that subwords consistently outperform characters across multiple benchmarks and language pairs.

Methodology and Experimental Setup

The study utilizes the S-Transformer architecture, a variant of the Transformer model optimized for speech. This architecture incorporates 2D convolutional layers to process audio spectrograms, reducing their dimensionality before they reach the attention layers.

To ensure a fair comparison, the authors maintained a consistent model size (approximately 35M parameters) and identical training hyperparameters. The primary variable was the target segmentation method:

  1. Character-level: The model predicts individual characters, resulting in long output sequences.
  2. Subword-level (BPE): The model predicts subword units based on 8,000 merge rules, resulting in significantly shorter sequences.

Datasets and Scope

The experiments were conducted across three major benchmarks to ensure robustness across different domains and data scales. The MuST-C corpus, in particular, allowed for testing across eight different language directions.

Table 1: Statistics of the datasets used in the study, showing the significant difference in sequence length between character and BPE representations.

Key Insights: The Subword Advantage

The researchers identified several reasons why subword segmentation provides superior results:

  • The Capacity Trap: Character-level models must dedicate a portion of their internal capacity to learning how characters form words before they can effectively learn translation mappings. BPE models bypass this, allowing more capacity for translation logic.
  • Overconfidence and Overfitting: Analysis of output distributions revealed that character-level models tend to be "peaked" or overconfident. This suggests they are prone to memorizing specific training sequences rather than learning generalizable linguistic patterns.
  • Sequence Length Challenges: Character-level models struggle as sentence length increases. Because character sequences are 3–4 times longer than BPE sequences, the model must track dependencies over a much larger number of time steps, which increases the difficulty of the translation task.
  • Multilingual Synergy: In multilingual settings where one model translates into multiple languages, the advantage of subwords becomes even more pronounced, likely due to better management of shared linguistic spaces.

Results and Performance

Subword-level models outperformed character-level models in every experimental setting. On the MuST-C benchmark, BPE models achieved gains of 1.2 to 3.3 BLEU points over their character-level counterparts. In the context of machine translation, a 3-point BLEU improvement is considered highly significant.

Table 2: Comparison of BLEU scores against previous state-of-the-art works, highlighting the performance of the BPE-based model.

Beyond accuracy, BPE models proved to be more efficient, offering faster training and decoding times due to the reduced sequence lengths. Qualitative analysis further showed that while character models might perform adequately on short, simple sentences, BPE models are far superior at handling complex terminology and long-form translations.

Conclusion

This work provides a definitive answer for researchers in direct speech translation: subword segmentation is the superior choice for target text representation. While character-level modeling remains relevant in ASR, the added complexity of translation makes the efficiency and semantic grouping of BPE indispensable for high-quality ST systems.

Summary updated

BibTeX
@inproceedings{gangi2020target,
  author = {Mattia Antonino Di Gangi and Marco Gaido and Matteo Negri and Marco Turchi},
  title = {{On Target Segmentation for Direct Speech Translation}},
  booktitle = {AMTA},
  year = {2020},
  url = {https://openreview.net/forum?id=4jflMftaH-R},
}
Figure