Abstract
Recently, deep reasoning large language models (LLMs) like DeepSeek-R1 have made significant progress in tasks such as mathematics and coding. Inspired by this, several studies have employed reinforcement learning (RL) to enhance models' deep reasoning capabilities and improve machine translation (MT) quality. However, terminology translation—an essential task in MT—remains unexplored in deep reasoning LLMs. In this paper, we propose TAT-R1, a terminology-aware translation model trained with reinforcement learning and word alignment. Specifically, we first extract keyword translation pairs using a word alignment model. Then, we carefully design three types of rule-based alignment rewards with the extracted alignment relationships. With those alignment rewards, the RL-trained translation model can learn to focus on the accurate translation of key information, including terminology in the source text. Experimental results show the effectiveness of TAT-R1. Our model significantly improves terminology translation accuracy compared to baseline models while maintaining comparable performance on general translation tasks. In addition, we conduct detailed ablation studies of the DeepSeek-R1-like training paradigm for machine translation and reveal several key findings.
Introduction
Machine Translation (MT) has achieved remarkable fluency in general contexts, yet it consistently struggles with domain-specific terminology. Technical nouns in legal, medical, or engineering texts often require precise mapping that standard models miss. Traditional solutions like Retrieval-Augmented Generation (RAG) or constrained decoding provide external guidance but do not improve the model's internal reasoning regarding lexical choices.
TAT-R1 addresses this by adapting the training paradigm of reasoning models like DeepSeek-R1 to the translation task. By rewarding the model for "thinking" about specific keywords before generating a translation, TAT-R1 achieves significant gains in terminology accuracy without sacrificing the natural flow of general translation.
Methods
TAT-R1 utilizes Group Relative Policy Optimization (GRPO), an RL algorithm that allows the model to learn from its own outputs by comparing a group of candidate translations. The primary innovation lies in the Word Alignment Reward system, which uses the unsupervised tool SimAlign to map correspondences between source and target words.
As shown in Figure 1, the model is guided by three specific alignment rewards:
- Answer-align-word reward ($R_{aaw}$): Rewards the inclusion of key nouns (identified via Named Entity Recognition) in the final answer.
- Answer-align-order reward ($R_{aao}$): Rewards placing those key terms in the correct relative order.
- Think-align-word reward ($R_{taw}$): A novel reward that encourages the model to mention correct key terms inside its
<think>tags before providing the final translation.
These are combined with a Format Reward to ensure structural integrity and a COMET Reward to maintain overall semantic quality.
Experimental Results
The model was trained on 16,000 samples across Chinese-English parallel data and evaluated on WMT23/24 and the RTT terminology benchmark.
Key Findings
- RL Superiority over SFT: Standard Supervised Fine-Tuning (SFT) can degrade performance if training data is noisy. RL allows the model to explore translations superior to imperfect reference texts.
- Functional Reasoning: By rewarding alignment within the
<think>block, the researchers forced the model to use that space for "lexical planning," directly improving the final output. - The BLEU Trap: The study confirms that optimizing solely for BLEU can lead to "reward hacking," where models produce stiff, unnatural text. Word alignment rewards provide a more flexible and human-aligned alternative.
- Zero-Shot Generalization: Despite being trained primarily on Chinese/English, the model's terminology reasoning capabilities transferred effectively to English-to-German tasks, outperforming the base Qwen model and models trained only with semantic rewards.
Conclusion
TAT-R1 demonstrates that reinforcement learning with word alignment is a powerful tool for solving terminology challenges in MT. While the current reasoning process is a one-pass planning phase, future work will explore self-correction mechanisms where the model recognizes and fixes its own errors within the reasoning block. Additionally, scaling these alignment rewards to larger models and more diverse language pairs remains a promising avenue for research.
Summary updated