Authors: Marco Gaido, Susana Rodríguez, Matteo Negri, Luisa Bentivogli, Marco Turchi.
Source: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Abstract
This paper introduces NEuRoparl-ST, the first benchmark specifically designed to evaluate how speech translation (ST) systems handle named entities and terminology. The authors find that while modern systems are decent at technical terms, they fail significantly on person names—achieving only 37-40% accuracy—highlighting a major gap in current end-to-end and cascade models.
Introduction
Automatic translation systems often struggle with "rare words," specifically Named Entities (NEs) (e.g., "Moby Dick") and domain-specific terminology (e.g., "plastic explosive"). In text-based translation, these are challenging; in speech translation, they present a significant hurdle. Speech lacks the orthographic cues, such as capitalization, that help text-based models identify entities. For instance, if an Automatic Speech Recognition (ASR) system outputs "moby dick" in lowercase, a translation engine might interpret it literally. Despite their importance for meaning, the field lacked a standardized benchmark to measure how well ST systems preserve these critical words.
Methodology and Benchmark
The authors introduce NEuRoparl-ST, a novel benchmark derived from the Europarl-ST corpus (European Parliament speeches). It covers English to Spanish, French, and Italian. The transcripts and translations were manually annotated with 18 types of Named Entities (following OntoNotes guidelines) and technical terms (using the IATE term base).
The study compares two dominant architectures:
- Cascade Models: A pipeline where an ASR model transcribes audio to text, which is then processed by a "true-caser" and fed into a Machine Translation (MT) model.
- Direct Models: A single end-to-end neural network mapping audio directly to target language text, trained using data augmentation and knowledge distillation from a pre-trained MT teacher.
Key Results
General performance analysis shows that ST systems correctly translate 75–80% of technical terms but only 65–70% of Named Entities. As shown in the table below, while MT models (translating from gold text) perform well, both Cascade and Direct ST models see a significant drop in accuracy for NEs across all language pairs.
The Person Name Pitfall
A critical finding is the "Person Name Pitfall." Person names are the hardest category for all systems. Unlike locations, which are frequent and have standard translations, person names have high acoustic variety and often lack context. Accuracy on person names drops to a staggering 37–40%, even though translating them from text is trivial (near 94% accuracy). This confirms that the bottleneck is the speech-to-text transition.
Comparative Insights
- Acoustic vs. Data Advantage: Direct models show a slight edge in handling NEs because they have "unmediated access" to the audio, avoiding error propagation from ASR (e.g., mishearing a name as a common noun). Conversely, Cascade models outperform direct models on terminology (+3.5% accuracy) because their MT component is trained on much larger text-only corpora.
- Casing Issues: In cascade systems, errors often stem from the "true-caser" failing to capitalize entities, which subsequently confuses the downstream MT model.
Conclusion
The NEuRoparl-ST benchmark reveals that current speech translation systems have a major gap in handling named entities, particularly person names. Future research should focus on improving acoustic modeling for rare names and integrating external knowledge sources, such as dictionaries or knowledge graphs, directly into end-to-end models to bridge the terminology and entity gap.
Summary updated