Abstract
We study the problem of domain adaptation in Neural Machine Translation (NMT) when domain-specific data cannot be shared due to confidentiality or copyright issues. As a first step, we propose to fragment data into phrase pairs and use a random sample to fine-tune a generic NMT model instead of the full sentences. Despite the loss of long segments for the sake of confidentiality protection, we find that NMT quality can considerably benefit from this adaptation, and that further gains can be obtained with a simple tagging technique.
Introduction
High-quality Neural Machine Translation (NMT) often requires fine-tuning on domain-specific data. However, in sensitive sectors such as legal or medical services, data owners—including law firms and pharmaceutical companies—are frequently restricted from sharing full translation memories due to strict confidentiality agreements, privacy regulations, or copyright concerns.
The central challenge addressed in this research is whether NMT models, which typically rely on long-range context provided by full sentences, can still benefit from adaptation when provided only with fragmented, shuffled data. If a data owner only releases short, disconnected phrase pairs to protect the structural integrity of original documents, can the model still learn meaningful domain-specific mappings?
Methodology
The proposed workflow balances data utility with privacy by transforming confidential documents into fragmented phrase pairs. The process involves four key stages:
- Phrase Extraction: Using statistical machine translation techniques, the researchers extract short sequences of words (phrase pairs) that maintain translation equivalence between the source and target languages.
- Fragmentation and Subsampling: To prevent the reconstruction of original documents, extracted phrases are shuffled, and a significant portion (50%) is discarded. This process effectively removes the "connective tissue" of the original text.
- Phrase Tagging: To prevent the model from becoming biased toward generating only short fragments, a tagging technique is introduced. Fragments are wrapped in
<P> ... </P>tags during training, signaling to the model that it is processing a fragment rather than a complete sentence. - Fine-Tuning: A state-of-the-art Transformer model, pre-trained on general news data, is fine-tuned using these tagged phrase pairs.
Experimental Setup and Results
The researchers evaluated this approach across three distinct domains: medical (EMEA), software documentation (GNOME), and legal text (JRC-Acquis). The baseline was a top-performing WMT19 news translation model.
The results demonstrate that terminology is a significant component of domain adaptation. In the software domain (GNOME), where specific vocabulary is paramount, phrase-based adaptation achieved a BLEU score of 37.0, remarkably close to the 38.9 achieved with full confidential sentences. In the medical domain (EMEA), the baseline of 35.5 was improved to 40.5 using tagged fragments.
However, the legal domain (JRC) proved more challenging. While the baseline improved slightly from 29.0 to 30.0, it remained far below the 54.7 BLEU score possible with full sentences. This suggests that in domains with highly complex, long-range syntactic structures, fragmentation may discard too much critical information.
Discussion and Conclusion
A key insight from this study is the effectiveness of the tagging technique. By explicitly marking fragments, the model avoids the "brevity penalty" and retains its ability to generate fluent, full-length sentences during inference. Furthermore, the study found that increasing phrase length (e.g., from 4 to 7 words) does not always improve performance, likely due to increased redundancy in the training data.
While this method provides a practical path for domain adaptation under data constraints, it highlights a structural gap in complex domains like law. Future research will explore formal privacy frameworks like Differential Privacy to quantify the trade-off between confidentiality and translation utility more precisely.
Summary updated