TL;DR
This paper introduces a new, large-scale English-Japanese Simultaneous Interpretation (SI) corpus, featuring data from interpreters with varying experience levels. By analyzing this corpus, the researchers found that more experienced interpreters better manage latency and maintain quality, and that high latency negatively impacts SI quality.
What Problem Does This Paper Address?
Simultaneous Interpretation (SI) is a challenging real-time translation task where interpreters translate while the speaker is still talking. While automatic speech translation has seen significant progress, high-quality corpora for studying and building SI systems, especially for the English-Japanese language pair, are scarce. Existing SI corpora are often limited in size or lack detailed information about interpreter experience. This paper aims to address this gap by constructing a substantial SI corpus and conducting a detailed analysis of interpreter performance based on experience, latency, and quality.
Methods
The researchers constructed a new English-Japanese SI corpus by recording simultaneous interpretations of lectures and press conferences. A key aspect of their methodology was to include SI data from three interpreters with different levels of experience (ranked S, A, and B based on years of experience). For a subset of the data, they performed manual sentence-level alignment between the source speech and the SI output. This allowed for a detailed analysis of three crucial SI aspects:
- Latency: Measured using Ear-Voice Span (EVS), which quantifies the delay between the original utterance and its interpretation.
- Quality: Evaluated using two metrics:
- BERTScore: An automatic metric that leverages contextualized subword embeddings to assess semantic similarity between the SI output and offline translations.
- Bunsetsu-level Semantic Preservation Score (BSPS): A novel metric developed by the authors that evaluates how well individual semantic units (bunsetsus) in the translation are conveyed in the SI output.
- Word Order: Analyzed using Kendall's K distance to compare the word order differences between the SI output and offline translations.
Datasets
The corpus was constructed using the following materials:
- TED Talks: For English to Japanese interpretation.
- TEDx Talks: For Japanese to English interpretation.
- Corpus of Spontaneous Japanese (CSJ): For Japanese to English interpretation.
- Japan National Press Club (JNPC) conferences: For Japanese to English interpretation.
The paper also utilized offline translations (Japanese subtitles) as a reference for evaluating SI quality and word order.
Key Insights and Ideas
- Experience Matters: The study strongly suggests that interpreter experience is a significant factor in SI performance. More experienced interpreters (S-rank) demonstrated better control over latency and maintained higher quality.
- Latency-Quality Trade-off Confirmed: The research reinforces the well-known trade-off between latency and quality in SI. High latency, particularly at the start of a sentence (EVSstart), was found to negatively impact the quality of the interpretation for that sentence.
- Novel Quality Metric (BSPS): The introduction of the Bunsetsu-level Semantic Preservation Score (BSPS) offers a more granular way to evaluate SI quality by focusing on the preservation of semantic units, complementing automatic metrics like BERTScore.
- Interpreter Strategies: The analysis revealed that interpreters, especially those with less experience, might employ strategies like summarization or generalization to manage latency, which can sometimes be a challenge for automatic evaluation metrics.
Key Results
- Latency: A-rank interpreters exhibited the largest latency, followed by B- and S-rank interpreters. Latency ranged from 2 to 4 seconds, with some instances exceeding 5 seconds.
- Quality (BERTScore): S-rank interpreters achieved the highest BERTScore (Precision, Recall, and F1), indicating superior quality compared to A- and B-rank interpreters.
- Quality (BSPS): Higher-ranked interpreters generally achieved higher BSPS scores, suggesting better preservation of semantic units.
- Word Order: The study observed that interpreters often adopted strategies to maintain the word order of the source language, especially when the target language (Japanese) has a different typical word order (SOV vs. English SVO).
- Drop/Skip Ratios: S-rank interpreters had lower "drop" ratios (segments not interpreted) compared to A- and B-rank interpreters.
Limitations and Future Work
- Limited Data for Analysis: While the corpus is large, the detailed sentence-level alignment and analysis were performed on a subset of 14 TED talks.
- Subjectivity in Metrics: While BERTScore and BSPS are valuable, the paper acknowledges the need for further research into their correlation with human evaluations, especially for specific types of talks or interpreter strategies.
- Human Evaluation: The authors suggest future work involving human evaluation by simultaneous interpreters themselves, rather than just translators, to gain deeper insights.
- Generalizability: The study focused on English-Japanese SI. Future work could explore these analyses across different language pairs.
- Automatic Evaluation of Latency: The paper mentions a simplified EVS calculation due to data limitations, suggesting potential for more sophisticated latency analysis.
Summary updated