Pretraining Language- and Domain-Specific BERT on Automatically Translated Text

Tatsuya Ishigaki, Yui Uehara, Goran Topic, Hiroya TakamuraRanlp 2023

Introduction

The effectiveness of pretrained language models like BERT is significantly enhanced when they are adapted to specific domains. However, while English-language scientific literature is abundant, many other languages, including Japanese, lack the large-scale, publicly available domain-specific corpora required for high-quality pretraining. This "resource gap" hinders the development of specialized NLP tools for researchers working in non-English contexts.

This paper investigates whether machine-translated (MT) text can serve as a viable substitute for human-authored text during the pretraining phase. By focusing on the materials science domain in Japanese, the authors demonstrate a framework for synthesizing a domain-specific corpus from English abstracts to train a high-performing BERT model.

Overview of the proposed framework: English domain-specific text is translated into Japanese to pretrain a BERT model.

Methodology

The researchers proposed a three-stage framework to bridge the data gap using Machine Translation:

  1. Data Acquisition & Translation: The team extracted approximately 2.5 million English abstracts (comprising 21 million sentences) from the Web of Science database, specifically targeting the "Materials Science" category. These were translated into Japanese using Amazon Translate.
  2. Vocabulary Construction: Recognizing that subword tokenization is critical for domain performance, the authors tested four vocabulary strategies: GeneralVocab (Wikipedia), TransVocab (translated text), MixVocab (mixture), and ConcatVocab (union of general and translated vocabularies).
  3. Pretraining Strategies: They compared different training regimes to determine the most efficient path to domain expertise.
Vocabulary construction methods and pretraining strategies compared in the study.

The pretraining strategies included TransTrain (training only on translated data), MixTrain (training on a mixture of Wikipedia and translated data), and PipelineTrain (continuing the pretraining of a general Wikipedia-based model using the translated domain data).

Experimental Setup and Evaluation

The models were evaluated on a manually annotated dataset of 27,053 sentences from Japanese journal papers (specifically Transactions of the Society of Polymer Science, Japan). The evaluation focused on two core tasks: Named Entity Recognition (NER) and Relation Extraction (RE) for properties such as "glass transition temperature" and "elasticity."

The pretraining data consisted of:

  • Domain Data: 2.5 million translated abstracts from the Web of Science.
  • General Data: A standard dump of Japanese Wikipedia (~1.2 million articles).

Key Results and Insights

The study evaluated 10 different model configurations against a baseline general Japanese BERT. The results consistently favored the models pretrained on translated text.

Performance Gains

Every configuration utilizing translated data outperformed the general BERT baseline on downstream tasks. In the "glass transition temperature" NER task, the best-performing translated model achieved an F1 score of 92.14, surpassing the baseline's 90.24. For Relation Extraction in the "elasticity" task, F1 scores improved from 71.00 (baseline) to 73.80.

The Impact of Vocabulary and "Translationese"

A significant finding was the resilience of the models to "translationese"—the unnatural phrasing often found in machine-translated text. The domain-specific knowledge embedded in the translated abstracts proved more valuable than the linguistic "noise" was detrimental. Furthermore, the ConcatVocab strategy performed best, indicating that the model benefits from a vocabulary that encompasses both general linguistic structures and specialized scientific terminology.

Conclusion

This research demonstrates that machine translation is a powerful tool for overcoming data scarcity in domain-specific NLP for non-English languages. The PipelineTrain approach, in particular, offers a cost-effective method for practitioners to adapt existing general-purpose models to specialized fields without the need for massive, human-authored target-language corpora. Future work may extend this methodology to generative architectures such as GPT or BART and further investigate the correlation between translation quality and model performance.

Summary updated

BibTeX
@inproceedings{ishigaki2023pretraining,
  author = {Tatsuya Ishigaki and Yui Uehara and Goran Topic and Hiroya Takamura},
  title = {{Pretraining Language- and Domain-Specific BERT on Automatically Translated Text}},
  booktitle = {Ranlp},
  year = {2023},
}
Figure