Abstract
Huawei’s Translation Service Center (HW-TSC) secured leading results at the CCMT 2024 competition by combining a Deep Transformer-big architecture with a robust suite of advanced training strategies. Central to their success was a multi-stage pipeline involving Regularized Dropout (R-Drop), Bidirectional Training, and Transductive Ensemble Learning. Furthermore, the team introduced an Automatic Post-Editing (APE) tool powered by a fine-tuned Llama 2-13B model, which significantly enhanced translation quality in multi-domain tasks by synthesizing the best elements of multiple NMT hypotheses.
1. Introduction
Neural Machine Translation (NMT) continues to face challenges related to data scarcity in minority languages and the "domain gap" in specialized fields. While synthetic data generation via back-translation is a common remedy, it introduces noise that can compromise model integrity. HW-TSC addressed these issues through a layered approach of training optimizations and the integration of Large Language Models (LLMs) to refine final outputs.
2. Methodology
The core architecture utilized was a Deep Transformer-big model, consisting of a 25-layer encoder and a 6-layer decoder. The development process was divided into two primary phases: advanced NMT training and LLM-based post-editing.
2.1 Advanced NMT Training Pipeline
The team implemented a layered sequence of techniques to maximize the utility of both authentic and synthetic data. Key strategies included:
- Regularized Dropout (R-Drop): Enhances model consistency by minimizing the difference between different dropout iterations of the same input.
- Bidirectional Training (BiT): A pre-training phase where the model learns translation in both directions (source-to-target and target-to-source) simultaneously.
- Alternated Training (AT): A method to mitigate the "poisoning" effect of noisy synthetic data by alternating training steps between synthetic and clean, authentic bilingual data.
- Transductive Ensemble Learning (TEL): An advanced ensemble method where a strong model is fine-tuned on synthetic data generated by an ensemble of models specifically for the test set.
2.2 LLM-based Automatic Post-Editing (APE)
To further bridge the domain gap, HW-TSC employed a Llama 2-13B model fine-tuned via Low-Rank Adaptation (LoRA). This model served as a "super-editor," taking the source text, a specific instruction, and the top 10 hypotheses from the NMT system as input. By leveraging the reasoning capabilities of the LLM, the system synthesized these inputs into a single, high-fidelity translation.
3. Datasets and Experimental Setup
The team participated in several tracks, including English-Chinese (en-zh), Chinese-English (zh-en), and minority languages (Mongolian, Tibetan, and Uyghur to Chinese). For the multi-domain task (zh-en), they utilized a custom "HypoTranslate" dataset. This dataset was curated by selecting high-quality bilingual pairs using the COMET-Kiwi metric and generating N-best hypotheses from their base NMT model to train the APE component.
4. Results and Analysis
The incremental application of these strategies yielded substantial improvements in translation quality:
- Bilingual (en→zh): The baseline BLEU score of 54.37 improved to 57.71 after the full NMT pipeline was applied.
- Multi-domain (zh→en): The most significant gains were observed here. While the strong TEL ensemble achieved a BLEU of 36.67, domain-specific fine-tuning raised this to 44.42. The addition of the Llama 2-based APE model further pushed the score to 47.33, representing a total improvement of nearly 11 points.
- Minority Languages: Techniques like Tagged Back-Translation proved highly effective for low-resource pairs such as Tibetan-to-Chinese (ti→zh) and Uyghur-to-Chinese (uy→zh).
5. Conclusion
The HW-TSC submission demonstrates that while architectural depth is important, the cumulative effect of specialized training strategies and LLM-based refinement is the key to state-of-the-art NMT. The use of LLMs as refiners rather than direct translators allows for higher fidelity by grounding the model in multiple hypotheses. Future work will focus on optimizing the computational efficiency of the APE stage and exploring its generalizability across more diverse language pairs.
Summary updated