TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

Julien Knafou, Luc Mottin, Anaïs Mottaz, Alexandre Flament, Patrick RuchEMNLP 2025

TL;DR

This paper introduces TransBERT, a novel framework for training domain-specific language models (DS-LMs) using synthetically translated text, addressing the scarcity of data in specialized domains. They demonstrate that this approach can achieve state-of-the-art performance on downstream tasks, even in low-resource settings, by releasing a comprehensive toolkit (TransCorpus), a large French life sciences corpus (TransCorpus-bio-fr), and a pre-trained model (TransBERT-bio-fr).

What Problem Does This Paper Address?

Natural Language Processing (NLP) models, particularly Pre-trained Language Models (PLMs), have revolutionized many fields. However, their success is heavily dependent on the availability of large, high-quality datasets. This is a significant problem for specialized domains (like life sciences, legal, finance) and for languages other than high-resource ones (like English, French). In these "low-resource" scenarios, the lack of data severely limits the development of effective NLP tools. The authors specifically highlight the challenge of building DS-LMs for languages like French in the life sciences domain, where existing models are either not domain-specific or are trained on relatively small datasets.

Methods

The core of the paper is the TransBERT framework, which leverages synthetic translation to create large domain-specific corpora. Instead of relying on human-translated data, which is expensive and scarce, they use machine translation (MT) to generate parallel text.

Key methodological components include:

  • TransCorpus Toolkit: A scalable toolkit built on the fairseq translation framework. It's designed to generate extensive synthetic domain-specific corpora in up to 100 languages. It features a production-level API/CLI, multi-GPU support, and checkpoint recovery for robust corpus management.
  • Machine Translation Model: They used M2M-100 (specifically the 1.2B parameter version) for its ability to translate directly between many language pairs without an English intermediary. They explored both sentence-based and document-based translation, finding sentence-based translation to be more efficient and scalable.
  • TransTokenizer: A custom tokenizer trained on the synthetically translated corpus. This is crucial for domain adaptation, as it can capture specific linguistic features of the target domain better than general-purpose tokenizers.
  • Pre-training: The TransBERT model is pre-trained using a BERT-like architecture (12 layers, 12 attention heads, 768 dimensions) following the RoBERTa methodology, with a large batch size (8k) and an Adam optimizer.

Datasets

  • Training Data: The primary training data is TransCorpus-bio-fr, a synthetically translated French life sciences corpus derived from the English MEDLINE/PubMed Baseline Repository (MBR). This corpus is substantial, containing 221 million sentences and 5.25 billion words, totaling 36.4GB of raw text.
  • Evaluation Data: The models were evaluated on DrBenchmark, a French life sciences benchmark comprising 15 datasets across classification, Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Semantic Textual Similarity (STS) tasks. This benchmark was adapted to include Hyperparameter Optimization (HPO) and k-fold cross-validation for more robust evaluation.

Key Insights and Ideas

  • Synthetic Data is Viable for SOTA: The most significant insight is that synthetically translated data, when used for pre-training, can achieve state-of-the-art performance in specialized domains, even for low-resource language pairs. This challenges the assumption that only high-quality, human-annotated data is sufficient.
  • Domain-Specific Tokenization Matters: Using a tokenizer trained on the target domain's synthetic data (TransTokenizer) leads to better performance compared to using a general-purpose tokenizer (CamemBERT's). This highlights the importance of tailoring linguistic representations to the specific domain.
  • Sentence-Based Translation Efficiency: For large-scale corpus generation, sentence-based translation with M2M-100 proved more efficient and scalable than document-based translation, despite potential issues like repetition in abstract-level translation.
  • Quality vs. Quantity: While DrBERT used a larger native corpus, TransBERT-bio-fr, despite being synthetically generated, was significantly larger and led to better performance, suggesting that the scale and domain-specificity of synthetic data can outweigh the size of smaller native corpora.

Key Results

  • Performance: TransBERT-bio-fr outperformed existing state-of-the-art models (CamemBERT and DrBERT) on 10 out of 15 DrBenchmark datasets, achieving statistically significant improvements in classification and NER tasks.
  • NER Advantage: TransBERT showed a notable advantage in Named Entity Recognition (NER) tasks, suggesting that domain-specific tokenization has a strong influence on this task.
  • Corpus Size: TransCorpus-bio-fr is approximately five times larger than the DrBERT corpus.
  • Translation Speed: The M2M-100 1.2B model with sentence-based translation was chosen for its balance of quality and speed.

Limitations and Future Work

  • Limitations:
    • Generalizability to Other Domains: While the framework shows promise, its effectiveness in other specialized domains (legal, finance, etc.) needs further investigation, as these domains may have unique linguistic challenges and cultural contexts.
  • MT System Dependency: The quality of synthetic data is heavily dependent on the quality of the underlying MT systems, which can vary significantly for different language pairs.
  • Linguistic Gaps: Significant linguistic differences between source and target languages can affect translation quality and introduce biases.
  • "Best" Model Fallacy: The authors acknowledge that a single "best" model is unlikely, and performance can vary based on context, task, and data distribution.
  • Future Work:
    • Expand Language Coverage: Extend the framework to more languages, especially underrepresented ones.
  • Multilingual Models: Develop multilingual DS-LMs for the life sciences sector to leverage cross-lingual knowledge transfer.
  • Generative LLMs for Data Augmentation: Explore using generative LLMs to create synthetic data as an alternative to translation.
  • Comparison with Large Language Models (LLMs): Conduct extensive comparisons with the latest generative LLMs to understand trade-offs between specialized and general models.

Summary updated

BibTeX
@inproceedings{knafou2025transbert,
  author = {Julien Knafou and Luc Mottin and Anaïs Mottaz and Alexandre Flament and Patrick Ruch},
  title = {{TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure