SJTU System Description for the WMT24 Low-Resource Languages of Spain Task

Tianxiang Hu, Haoxiang Sun, Ruize Gao, Jialong Tang, Pei Zhang, Baosong Yang, Rui WangWMT 2024 · DOI

Introduction

The WMT24 Low-Resource Languages of Spain task focuses on translating Spanish into three regional languages: Aragonese (spa-arg), Aranese (spa-arn), and Asturian (spa-ast). These language pairs present a significant challenge due to the extreme scarcity of high-quality parallel corpora. Furthermore, the evaluation sets for this task are derived from the outputs of Apertium, a rule-based machine translation system, which were subsequently post-edited by humans. This creates a specific "Apertium bias," where models must not only be linguistically accurate but also align with the stylistic choices of the rule-based engine to achieve high scores.

As shown in Figure 1, standard large language models (LLMs) like GPT-4 and Llama 3 struggle to compete with the specialized Apertium system on these specific pairs without targeted adaptation.

Comparison of rule-based Apertium vs. standard LLMs.

Methodology

To address these challenges, the SJTU team developed a multi-stage pipeline centered on "distilling" the knowledge of the rule-based system into a compact neural model.

1. Forward Distillation

The team utilized the Apertium system as a "teacher" to translate 1 million Spanish sentences from the NLLB (No Language Left Behind) corpus into the three target languages. This generated a large-scale synthetic pseudo-corpus that captures the specific translation patterns of Apertium.

2. Model Fine-Tuning

A Qwen2-0.5B model was fine-tuned on this synthetic data. Despite its small size, this model was able to effectively clone the behavior of the Apertium system, bridging the gap between rule-based precision and neural flexibility.

3. Refinement Strategies

To further optimize performance, the team explored several advanced techniques:

  • 5-shot Supervised Fine-Tuning (SFT): Fine-tuning on FLORES+ development sets to enhance in-context learning.
  • Contrastive Preference Optimization (CPO): This technique was used to teach the model to prefer human-corrected versions over raw Apertium outputs, effectively learning to fix common rule-based errors.
  • Retrieval-Augmented Prompting: For the unconstrained track, the team used BM25 retrieval to provide Claude 3.5-Sonnet with the most relevant translation examples from the development set.

Results and Analysis

The distillation approach proved highly effective. The tiny 0.5B model, after distillation, was able to match or exceed the performance of models 100 times its size.

Final performance across different refinement strategies.

As detailed in Figure 4, the optimal strategy varied by language pair:

  • Aragonese: Distillation combined with 5-shot SFT achieved the highest score of 69.9 BLEU.
  • Aranese: The CPO approach was most effective, reaching 41.4 BLEU.
  • Asturian: A retrieval-augmented approach using Claude 3.5-Sonnet with 20-shot BM25 yielded the best result of 25.2 BLEU.

The team also found that Forward Translation (FT) was significantly more effective than Back Translation (BT) in this context. This is because FT directly mimics the target-side style required by the evaluation metrics, whereas BT fails to capture the specific "Apertium flavor."

Conclusion

The SJTU system demonstrates that for niche, low-resource tasks with specific stylistic requirements, distilling specialized rule-based systems into small LLMs is a powerful strategy. By combining synthetic data generation with refinement techniques like CPO and retrieval-augmented prompting, the team significantly outperformed general-purpose massive models. Future work will focus on improving data quality and exploring how these models perform on natural text outside the Apertium-influenced evaluation framework.

Summary updated

BibTeX
@inproceedings{hu2024sjtu,
  author = {Tianxiang Hu and Haoxiang Sun and Ruize Gao and Jialong Tang and Pei Zhang and Baosong Yang and Rui Wang},
  title = {{SJTU System Description for the WMT24 Low-Resource Languages of Spain Task}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.92},
}
Figure