TL;DR
AppTek presents their approach to the IWSLT 2023 subtitling and formality tracks, achieving competitive results in speech translation for subtitles and demonstrating effective formality control in machine translation. Their subtitling system, particularly the cascaded approach, significantly outperformed other participants, while their formality control system showed near-perfect accuracy for Russian and strong performance for Portuguese.
What Problem Does This Paper Address?
The paper tackles two key challenges in speech translation: creating high-quality subtitles from spoken audio and controlling the stylistic formality of machine translations. Subtitling requires not only accurate transcription and translation but also precise segmentation of speech into readable chunks with appropriate timing. This involves handling various audio conditions, from clear speech to noisy environments with music and multiple speakers. Furthermore, in many real-world applications, the formality of the translated text is crucial for appropriate communication. This paper addresses the need for systems that can handle both the technical demands of subtitling and the nuanced requirement of stylistic formality in translation.
Methods
AppTek employed a multi-component pipeline for speech translation and subtitling. For audio segmentation, they utilized the SHAS method, experimenting with different thresholds to optimize performance across various domains. Their speech translation (ST) pipeline involved both direct end-to-end models and cascaded approaches (ASR followed by MT). They trained Conformer-Transformer based models for ASR and Transformer Big models for MT. A key aspect of their approach was the use of text normalization (NEWTN) to handle ASR output lacking punctuation and casing, and length control mechanisms like length ROVER to ensure translations fit subtitle constraints. For subtitle creation, they explored both a "source template" approach, where source language subtitles guide target subtitle creation, and a "template-free" approach for direct speech translation.
For the formality track, AppTek leveraged their existing production MT systems, which support style control via pseudo-tokens. These systems were trained on large parallel corpora partitioned into formal, informal, and "no style" classes using regular expressions.
Datasets
The paper utilized a wide array of publicly available datasets for training, including:
- Text-parallel data: Europarl, News Commentary, CORDIS News, Tatoeba, TED2020, MuST-C, CoVoST v2, and OpenSubtitles.
- Speech data: Europarl-ST, How2, MuST-C, TED-LIUM, LibriSpeech, Mozilla Common Voice, VoxPopuli, CoVOST, and IWSLT TED.
- Direct Speech Translation data: Europarl-ST, MuST-C, CoVOST, and IWSLT TED.
Synthetic data was generated by translating English transcripts of speech data using their MT models. For the formality track, they used large public data from the OPUS collection.
Key Insights and Ideas
- Cascaded systems excel in subtitling: The paper highlights that cascaded ASR + MT systems generally outperform direct end-to-end ST systems for subtitling, especially when combined with effective length control.
- Audio segmentation is critical and domain-dependent: The choice of audio segmentation parameters (like the pDAC threshold) significantly impacts translation quality, with different optimal settings for different domains (e.g., TV series vs. TED talks).
- Length control is vital for subtitles: Mechanisms like length ROVER are crucial for ensuring translations fit within subtitle constraints, leading to substantial improvements in quality metrics.
- Formality control is achievable with regex-based data partitioning: AppTek demonstrated that their formality control system, trained on data partitioned using regular expressions, can achieve high accuracy, though the effectiveness of the regexes themselves can be a limiting factor.
- Human references can be inconsistent: The paper notes that in the formality track, the BLEU score for informal translations was surprisingly higher than for formal ones, suggesting that human references might not always strictly adhere to the requested style.
Key Results
- Subtitling Track: AppTek's cascaded systems outperformed other participants by an absolute 5% on the SUBER subtitle quality metric. Their En-De cascaded system achieved SUBER scores ranging from 63.0% to 87.6% across different domains, while the En-Es cascaded system showed acceptable quality for TED and EPTV content.
- Formality Track: For En-Ru, their system achieved near-perfect formality level accuracy (>98%) with strong BLEU and COMET scores. For En-Pt, they achieved 99% formality accuracy for the formal style but only 64% for the informal style, attributing this to imperfect regular expressions.
Limitations and Future Work
- Direct ST model limitations: The direct ST model showed imperfections like repetitions and lacked length control, contributing to its lower performance compared to cascaded systems.
- Domain-specific challenges: The movie subtitle domain posed significant challenges for ASR, and the fitness video domain was difficult for MT.
- Speech segmentation for noisy audio: Improving speech segmentation for challenging audio conditions (music, background noise, multiple speakers) is a key area for future work.
- Formality rule creation: The reliance on manual regular expressions for formality data partitioning is a limitation. Future work aims to reduce or eliminate the need for writing these rules.
- Integrating length control into direct ST: The authors plan to incorporate length control into their direct ST models to improve performance in time-constrained applications.
Summary updated