Leveraging High-Resource English Corpora for Cross-lingual Domain Adaptation in Low-Resource Japanese Medicine via Continued Pre-training

Kazuma Kobayashi, Zhen Wan, Fei Cheng, Tsuta Yuma, Xin Zhao, Junfeng Jiang, Jiahao Huang, Zhiyi Huang, Yusuke Oda, Rio Yokota, Yuki Arase, Daisuke Kawahara, Akiko Aizawa, Sadao KurohashiEMNLP 2025 · OpenReview

Abstract

Limited low-resource language corpora in professional domains like medicine hinder cross-lingual domain adaptation of pre-trained large language models (PLMs). While abundant English medical corpora could complement this scarcity, the effective mixture of English and target language, including machine-translated content, remains underexplored. We examined how linguistic features (e.g., token sizes and language proportions) affect performance on a Japanese–English medical knowledge benchmark. Through continued pre-training of a bilingual PLM on multilingual corpora with varying proportions of English and Japanese texts (both original and machine-translated), we analyzed correlations between linguistic features and fine-grained task performance. Our findings suggest a practical approach to optimizing multilingual corpora for cross-lingual domain adaptation, which requires leveraging specialized knowledge from English corpora while ensuring sufficient coverage of language-specific expressions in a target language (Japanese). Such insights will contribute to the development of multilingual models that effectively leverage English-language resources in various professional domains with low-resource languages.

Introduction

In specialized fields like medicine, language resources are heavily imbalanced. While English-language databases like PubMed host over 38 million papers, Japanese counterparts like J-STAGE contain only about 5 million. This "data desert" makes it difficult to adapt Large Language Models (LLMs) to professional domains in non-English languages. The core challenge is not just a lack of data, but the absence of an optimal "recipe" for mixing original English, original Japanese, and machine-translated (MT) content during continued pre-training (CPT).

Methodology

The researchers systematically investigated how corpus composition affects a model's medical knowledge acquisition. Starting with a 13-billion-parameter bilingual (Japanese–English) model (LLM-jp), they performed continued pre-training using six different corpus "recipes" designed to test various language balances.

The three-step methodology: continued pre-training, evaluation on JMedBench, and partial correlation analysis.

To move beyond simple observations, the study employed Partial Correlation Analysis. This statistical technique allowed the team to isolate the impact of a single variable (such as "English token count") on task performance while holding other variables (like "Total token count") constant.

Corpus Composition and Datasets

The authors utilized the PubMed Central (PMC) Open Access Subset, J-STAGE, medical textbooks, and clinical guidelines. They also generated a massive machine-translated version of the PMC full-text subcorpus to supplement the Japanese data. The six experimental recipes were:

  • EnJa-Base: A baseline mix of textbooks and abstracts.
  • JaDominant: Adds machine-translated Japanese versions of PMC articles.
  • EnDominant: Adds the original English PMC articles instead.
  • EnJa-Plus: Adds original English plus a half-dose of translated Japanese.
  • EnJa-Balance: Adds original English plus a full dose of translated Japanese.
  • EnJa-Hybrid: Adds even more textbooks and parallel (translated) data.
Token counts and language proportions for the six experimental corpus recipes.

Evaluation was conducted using JMedBench, a comprehensive benchmark consisting of 27 tasks (20 in Japanese, 7 in English) covering Multiple-Choice Question Answering (MCQA), Named Entity Recognition (NER), Document Classification (DC), and Semantic Textual Similarity (STS).

Key Results and Insights

The EnJa-Balance corpus emerged as the optimal configuration, achieving an average score of 0.475 on Japanese tasks, outperforming both the English-only addition (0.468) and the Japanese-translation-only addition (0.453).

  • English as a Knowledge Powerhouse: Surprisingly, adding original English medical text often improved performance on Japanese tasks more than adding machine-translated Japanese text did. This suggests that LLMs are highly efficient at transferring "conceptual" medical knowledge across the language barrier.
  • The "Balance" Sweet Spot: While machine-translated data is often lower quality than original text, it provides an additive effect by helping the model map English medical concepts to Japanese linguistic structures.
  • Diminishing Returns: The "EnJa-Hybrid" model, despite having the highest token count, performed worse than "EnJa-Balance." This highlights that simply increasing data volume can be counterproductive if the mixture is not carefully tuned.
  • Statistical Significance: Partial correlation analysis confirmed that English token count had the strongest and most consistent positive correlation with performance across nearly all medical tasks, regardless of the target language.

Conclusion and Future Work

This study demonstrates that effective cross-lingual domain adaptation requires a strategic mix of high-resource English data and target-language translations. While English provides the core domain knowledge, balanced exposure to the target language ensures linguistic naturalness. Future research will address the computational costs of training large-scale models and explore the impact of varying translation qualities on knowledge transfer.

Summary updated

Abstract

Limited low-resource language corpora in professional domains like medicine hinder cross-lingual domain adaptation of pre-trained large language models (PLMs). While abundant English medical corpora could complement this scarcity, the effective mixture of English and target language, including machine-translated content, remains underexplored. We examined how linguistic features (e.g., token sizes and language proportions) affect performance on a Japanese–English medical knowledge benchmark. Through continued pre-training of a bilingual PLM on multilingual corpora with varying proportions of English and Japanese texts (both original and machine-translated), we analyzed correlations between linguistic features and fine-grained task performance. Our findings suggest a practical approach to optimizing multilingual corpora for cross-lingual domain adaptation, which requires leveraging specialized knowledge from English corpora while ensuring sufficient coverage of language-specific expressions in a target language (Japanese). Such insights will contribute to the development of multilingual models that effectively leverage English-language resources in various professional domains with low-resource languages.
BibTeX
@inproceedings{kobayashi2025leveraging,
  author = {Kazuma Kobayashi and Zhen Wan and Fei Cheng and Tsuta Yuma and Xin Zhao and Junfeng Jiang and Jiahao Huang and Zhiyi Huang and Yusuke Oda and Rio Yokota and Yuki Arase and Daisuke Kawahara and Akiko Aizawa and Sadao Kurohashi},
  title = {{Leveraging High-Resource English Corpora for Cross-lingual Domain Adaptation in Low-Resource Japanese Medicine via Continued Pre-training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=EmCIpuJAa0},
}
Figure