TL;DR
This paper introduces TransBERT, a novel framework for training domain-specific language models (DS-LMs) using synthetically translated text, addressing the scarcity of data in specialized domains. They demonstrate that this approach can achieve state-of-the-art performance on downstream tasks, even in low-resource settings, by releasing a comprehensive toolkit (TransCorpus), a large French life sciences corpus (TransCorpus-bio-fr), and a pre-trained model (TransBERT-bio-fr).
What Problem Does This Paper Address?
Natural Language Processing (NLP) models, particularly Pre-trained Language Models (PLMs), have revolutionized many fields. However, their success is heavily dependent on the availability of large, high-quality datasets. This is a significant problem for specialized domains (like life sciences, legal, finance) and for languages other than high-resource ones (like English, French). In these "low-resource" scenarios, the lack of data severely limits the development of effective NLP tools. The authors specifically highlight the challenge of building DS-LMs for languages like French in the life sciences domain, where existing models are either not domain-specific or are trained on relatively small datasets.
Methods
The core of the paper is the TransBERT framework, which leverages synthetic translation to create large domain-specific corpora. Instead of relying on human-translated data, which is expensive and scarce, they use machine translation (MT) to generate parallel text.
Key methodological components include:
- TransCorpus Toolkit: A scalable toolkit built on the
fairseqtranslation framework. It's designed to generate extensive synthetic domain-specific corpora in up to 100 languages. It features a production-level API/CLI, multi-GPU support, and checkpoint recovery for robust corpus management. - Machine Translation Model: They used M2M-100 (specifically the 1.2B parameter version) for its ability to translate directly between many language pairs without an English intermediary. They explored both sentence-based and document-based translation, finding sentence-based translation to be more efficient and scalable.
- TransTokenizer: A custom tokenizer trained on the synthetically translated corpus. This is crucial for domain adaptation, as it can capture specific linguistic features of the target domain better than general-purpose tokenizers.
- Pre-training: The TransBERT model is pre-trained using a BERT-like architecture (12 layers, 12 attention heads, 768 dimensions) following the RoBERTa methodology, with a large batch size (8k) and an Adam optimizer.
Datasets
- Training Data: The primary training data is TransCorpus-bio-fr, a synthetically translated French life sciences corpus derived from the English MEDLINE/PubMed Baseline Repository (MBR). This corpus is substantial, containing 221 million sentences and 5.25 billion words, totaling 36.4GB of raw text.
- Evaluation Data: The models were evaluated on DrBenchmark, a French life sciences benchmark comprising 15 datasets across classification, Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Semantic Textual Similarity (STS) tasks. This benchmark was adapted to include Hyperparameter Optimization (HPO) and k-fold cross-validation for more robust evaluation.
Key Insights and Ideas
- Synthetic Data is Viable for SOTA: The most significant insight is that synthetically translated data, when used for pre-training, can achieve state-of-the-art performance in specialized domains, even for low-resource language pairs. This challenges the assumption that only high-quality, human-annotated data is sufficient.
- Domain-Specific Tokenization Matters: Using a tokenizer trained on the target domain's synthetic data (TransTokenizer) leads to better performance compared to using a general-purpose tokenizer (CamemBERT's). This highlights the importance of tailoring linguistic representations to the specific domain.
- Sentence-Based Translation Efficiency: For large-scale corpus generation, sentence-based translation with M2M-100 proved more efficient and scalable than document-based translation, despite potential issues like repetition in abstract-level translation.
- Quality vs. Quantity: While DrBERT used a larger native corpus, TransBERT-bio-fr, despite being synthetically generated, was significantly larger and led to better performance, suggesting that the scale and domain-specificity of synthetic data can outweigh the size of smaller native corpora.
Key Results
- Performance: TransBERT-bio-fr outperformed existing state-of-the-art models (CamemBERT and DrBERT) on 10 out of 15 DrBenchmark datasets, achieving statistically significant improvements in classification and NER tasks.
- NER Advantage: TransBERT showed a notable advantage in Named Entity Recognition (NER) tasks, suggesting that domain-specific tokenization has a strong influence on this task.
- Corpus Size: TransCorpus-bio-fr is approximately five times larger than the DrBERT corpus.
- Translation Speed: The M2M-100 1.2B model with sentence-based translation was chosen for its balance of quality and speed.
Limitations and Future Work
- Limitations:
- Generalizability to Other Domains: While the framework shows promise, its effectiveness in other specialized domains (legal, finance, etc.) needs further investigation, as these domains may have unique linguistic challenges and cultural contexts.
- MT System Dependency: The quality of synthetic data is heavily dependent on the quality of the underlying MT systems, which can vary significantly for different language pairs.
- Linguistic Gaps: Significant linguistic differences between source and target languages can affect translation quality and introduce biases.
- "Best" Model Fallacy: The authors acknowledge that a single "best" model is unlikely, and performance can vary based on context, task, and data distribution.
- Future Work:
- Expand Language Coverage: Extend the framework to more languages, especially underrepresented ones.
- Multilingual Models: Develop multilingual DS-LMs for the life sciences sector to leverage cross-lingual knowledge transfer.
- Generative LLMs for Data Augmentation: Explore using generative LLMs to create synthetic data as an alternative to translation.
- Comparison with Large Language Models (LLMs): Conduct extensive comparisons with the latest generative LLMs to understand trade-offs between specialized and general models.
Summary updated