On Creating an English-Thai Code-switched Machine Translation in Medical Domain

Parinthapat Pengpun, Krittamate Tiankanon, Amrest Chinkamol, Jiramet Kinchagawat, Pitchaya Chairuengjitjaras, Pasit Supholkhan, Pubordee Aussavavirojekul, Chiraphat Boonnag, Kanyakorn Veerakanjana, Hirunkul Phimsiri, Boonthicha Sae-jia, Nattawach Sataudom, Piyalitt Ittichaiwong, Peerat LimkonchotiwatEMNLP 2024 · DOI

Abstract

This research introduces a specialized Machine Translation (MT) approach for the medical domain that incorporates "code-switching"—the practice of mixing English technical terms into Thai sentences. By fine-tuning the NLLB (No Language Left Behind) model on a custom-built dataset, the authors created a system that medical professionals significantly prefer over standard monolingual translations. The study presents a novel "Pseudo-translation Masking and Generation" pipeline and the first benchmark dataset for medical English-Thai code-switched translation.

Introduction

In the medical field, precision is a matter of life and death. However, standard machine translation tools often struggle with Thai medical contexts because many English technical terms lack direct Thai equivalents, or their "correct" translations are too obscure for clinical use. For example, a term like "hyperinflation" might be incorrectly translated using the Thai word for economic inflation, destroying the factual utility of the text.

Thai medical professionals naturally solve this by code-switching: maintaining technical English keywords while using Thai for the surrounding sentence structure. This paper aims to bridge the gap between how machines translate (monolingual) and how experts actually communicate (code-switched).

Methodology

To address the lack of existing code-switched training data, the authors proposed a "Pseudo-translation Masking and Generation" pipeline:

  1. Keyword Identification: GPT-4 identifies critical medical keywords (symptoms, drugs, anatomy) in English source sentences.
  2. Masking: These keywords are replaced with unique placeholders.
  3. Translation: The masked sentence is translated into Thai using a standard MT model.
  4. Re-insertion: The original English keywords are swapped back into the Thai sentence, creating a "pseudo-code-switched" (pseudo-CS) output.
Comparison of standard NMT vs. code-switched output (NLLB-1) for a medical description.

This method was used to generate a large-scale dataset to fine-tune several variants of the NLLB-200 model.

Datasets

The study introduces a new benchmark dataset specifically designed for medical English-Thai code-switched translation:

  • Training Set: Approximately 64,000 English-to-Thai CS translation pairs.
  • Test Set: 1,100 pairs, rigorously verified and corrected by human annotators and Medical Doctors (MDs) to ensure factual accuracy and natural flow.
  • Source Material: 10,000 medical excerpts focused on differential diagnosis and multiple-choice medical questions.

Results and Discussion

The researchers evaluated 52 different model configurations using both automated metrics and human evaluation from 23 Medical Doctors.

Human Preference and Factual Accuracy

Traditional MT metrics like BLEU and chrF correlate poorly with the preferences of medical professionals. While Google NMT often scores high on automated metrics, it frequently "over-translates" technical terms, leading to lower human preference scores compared to code-switched models.

Glicko MD scores showing human preference for various models, with NLLB-1 and GPT-4-CS performing strongly.

The fine-tuned NLLB-1 model achieved a Glicko rating of 1549.55, significantly outperforming Google NMT (1398.61) and GPT-3.5 (1316.40). In terms of factual accuracy, the GPT-4-CS + Mask configuration achieved the highest score of 6.000 on a 7-point scale.

The Power of Masking

A key insight is that simply preventing a model from attempting to translate specific keywords (the "Mask" approach) significantly boosts factual accuracy across almost all model architectures, including large language models like GPT-4.

Conclusion

This research demonstrates that code-switching is a viable and preferred strategy for medical machine translation in Thai. While limitations exist regarding prompt sensitivity and hyperparameter tuning, the results provide a strong foundation for future work. Expanding this methodology to other language pairs where technical code-switching is common, such as English-Arabic or English-Spanish, represents a promising direction for professional domain translation.

Summary updated

BibTeX
@inproceedings{pengpun2024creating,
  author = {Parinthapat Pengpun and Krittamate Tiankanon and Amrest Chinkamol and Jiramet Kinchagawat and Pitchaya Chairuengjitjaras and Pasit Supholkhan and Pubordee Aussavavirojekul and Chiraphat Boonnag and Kanyakorn Veerakanjana and Hirunkul Phimsiri and Boonthicha Sae-jia and Nattawach Sataudom and Piyalitt Ittichaiwong and Peerat Limkonchotiwat},
  title = {{On Creating an English-Thai Code-switched Machine Translation in Medical Domain}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.351},
}
Figure