Abstract
The HW-TSC team (Huawei) developed a high-performing cascade system for the IWSLT 2020 Video Speech Translation task. By focusing on a three-stage pipeline—Automatic Speech Recognition (ASR), a specialized BERT-based disfluency correction module, and a domain-aware Neural Machine Translation (NMT) model—they demonstrated that "cleaning" spoken language is just as important as the translation itself.
Introduction
Translating speech from video is notoriously difficult because spoken language is "messy." Unlike written text, speech is filled with "disfluencies"—filler words like "uh" or "you know," repetitions, and false starts. In a cascade system (where ASR happens first, followed by translation), these errors propagate. If the ASR outputs "I wanna wanna... go to the the AI lab," a standard translation model might struggle to produce a fluent sentence. While some researchers explore "end-to-end" models that go straight from audio to translated text, these require massive amounts of rare, high-quality aligned data. The authors tackle the challenge of making a modular cascade system robust enough to handle the noise of real-world speech.
System Architecture
The system architecture consists of three distinct modules:
- Proprietary ASR: A closed-source system used to extract text from video soundtracks.
- Disfluency Correction (The "Cleaner"): This is the paper's core innovation. It uses two sub-methods:
- BERT-based Tagging: They fine-tuned a BERT-base model to perform sequence tagging. The model looks at ASR output and labels each token as "OK" (keep) or "BAD" (delete).
- N-Gram Mining: A rule-based approach to catch high-frequency ASR errors. For example, it maps "a i" to "AI" or "r and d" to "R&D," which are common phonetic misinterpretations in technical contexts.
- Domain-Aware NMT: The translation engine is a "Transformer-Big" model. To handle training data from different sources (like news vs. UN proceedings), they added domain discrimination tags (e.g.,
[CWMT]) at the start of sentences. This allows the model to learn the specific "style" of different datasets simultaneously.
Datasets and Evaluation
The team utilized several datasets for training and evaluation:
- Disfluency Training: A dataset of ~200,000 tokens derived from internal meeting recordings, transcribed by humans and aligned with ASR output using the
sclitetool. - NMT Training: Approximately 26 million parallel sentence pairs from the WMT 2019 Chinese-to-English news translation task (6.5M from CWMT and 20M from UN corpora).
- Evaluation: The CCMT 2018 news translation test set was used to benchmark the NMT component.
Results and Analysis
The authors showed significant incremental improvements at each stage of the pipeline.
Disfluency Correction Performance
On their internal test set, the baseline ASR output had a Word Error Rate (WER) of 33.1. As illustrated in Figure 1, the addition of domain rules, the BERT model, and N-gram mining brought the WER down to 24.9—a substantial improvement in text quality before it even reached the translator.
Translation Quality
The NMT system achieved a BLEU score of 39.25 on the CCMT dataset. This was achieved by combining the Transformer-Big architecture with domain classification (+1.36 BLEU) and model ensembling (+1.62 BLEU), as detailed in Figure 2.
Discussion
- Visual Context is Optional: Interestingly, the authors chose to ignore the visual frames of the video entirely. They argue that for many translation tasks, visual context isn't strictly necessary, and treating it as an offline speech translation task is more efficient.
- The "Sclite" Labeling Trick: To train their BERT cleaner, they used the
sclitetool to compare ASR output against human transcripts. By identifying insertions and substitutions automatically, they could generate "OK/BAD" labels for the BERT model without manual tagging. - Cascade vs. End-to-End: The authors argue that the cascade approach is more feasible for production because it allows each module (ASR, Correction, NMT) to be trained and optimized independently on different types of data.
Limitations and Future Work
The ASR system is proprietary, making it difficult for outside researchers to replicate the exact front-end results. While they found visual context unnecessary for this specific task, future work could explore if "grounding" the translation in video frames could resolve ambiguities that audio alone cannot. Additionally, the current disfluency model focuses on deleting bad tokens but does not attempt to correct or rephrase them, which could be a limitation for more complex ASR errors.
Summary updated