Introduction
The WMT24 Low-Resource Languages of Spain task focuses on translating Spanish into three regional languages: Aragonese (spa-arg), Aranese (spa-arn), and Asturian (spa-ast). These language pairs present a significant challenge due to the extreme scarcity of high-quality parallel corpora. Furthermore, the evaluation sets for this task are derived from the outputs of Apertium, a rule-based machine translation system, which were subsequently post-edited by humans. This creates a specific "Apertium bias," where models must not only be linguistically accurate but also align with the stylistic choices of the rule-based engine to achieve high scores.
As shown in Figure 1, standard large language models (LLMs) like GPT-4 and Llama 3 struggle to compete with the specialized Apertium system on these specific pairs without targeted adaptation.
Methodology
To address these challenges, the SJTU team developed a multi-stage pipeline centered on "distilling" the knowledge of the rule-based system into a compact neural model.
1. Forward Distillation
The team utilized the Apertium system as a "teacher" to translate 1 million Spanish sentences from the NLLB (No Language Left Behind) corpus into the three target languages. This generated a large-scale synthetic pseudo-corpus that captures the specific translation patterns of Apertium.
2. Model Fine-Tuning
A Qwen2-0.5B model was fine-tuned on this synthetic data. Despite its small size, this model was able to effectively clone the behavior of the Apertium system, bridging the gap between rule-based precision and neural flexibility.
3. Refinement Strategies
To further optimize performance, the team explored several advanced techniques:
- 5-shot Supervised Fine-Tuning (SFT): Fine-tuning on FLORES+ development sets to enhance in-context learning.
- Contrastive Preference Optimization (CPO): This technique was used to teach the model to prefer human-corrected versions over raw Apertium outputs, effectively learning to fix common rule-based errors.
- Retrieval-Augmented Prompting: For the unconstrained track, the team used BM25 retrieval to provide Claude 3.5-Sonnet with the most relevant translation examples from the development set.
Results and Analysis
The distillation approach proved highly effective. The tiny 0.5B model, after distillation, was able to match or exceed the performance of models 100 times its size.
As detailed in Figure 4, the optimal strategy varied by language pair:
- Aragonese: Distillation combined with 5-shot SFT achieved the highest score of 69.9 BLEU.
- Aranese: The CPO approach was most effective, reaching 41.4 BLEU.
- Asturian: A retrieval-augmented approach using Claude 3.5-Sonnet with 20-shot BM25 yielded the best result of 25.2 BLEU.
The team also found that Forward Translation (FT) was significantly more effective than Back Translation (BT) in this context. This is because FT directly mimics the target-side style required by the evaluation metrics, whereas BT fails to capture the specific "Apertium flavor."
Conclusion
The SJTU system demonstrates that for niche, low-resource tasks with specific stylistic requirements, distilling specialized rule-based systems into small LLMs is a powerful strategy. By combining synthetic data generation with refinement techniques like CPO and retrieval-augmented prompting, the team significantly outperformed general-purpose massive models. Future work will focus on improving data quality and exploring how these models perform on natural text outside the Apertium-influenced evaluation framework.
Summary updated