TL;DR
This paper systematically evaluates how different prompting strategies affect the performance of multilingual Large Language Models (LLMs) for medical English-to-Vietnamese translation. They found that larger LLMs generally perform better, and while simple few-shot prompting offers minimal gains, incorporating domain-specific medical terminology and using embedding-based example retrieval significantly boosts translation accuracy and fidelity.
What Problem Does This Paper Address?
Accurate medical translation is critical for healthcare access, especially in low-resource languages like Vietnamese. However, medical texts present unique challenges: specialized terminology, abbreviations, and complex phrasing often trip up general-purpose translation models. While multilingual LLMs show promise, their effectiveness for medical translation, particularly for English-to-Vietnamese (En-Vi), hasn't been thoroughly investigated. Existing studies often lack controlled comparisons of prompting techniques and standardized evaluation protocols, especially concerning how to best handle crucial medical terminology. This paper aims to fill this gap by systematically evaluating various prompting strategies across different LLM sizes for medical En-Vi translation.
Methods
The researchers explored three main prompting strategies for medical En-Vi translation:
- Zero-shot prompting: The LLM translates without any examples.
- N-shot prompting: The LLM is given a small number (1, 8, 16, or 32) of randomly sampled English-Vietnamese sentence pairs from the training data as examples.
- Dictionary-augmented prompting: This is the core innovation. It involves using a curated medical lexicon (Meddict) to guide the translation. Two variations were tested:
- Zero-shot with terminology hints: Key medical terms and their Vietnamese equivalents are directly inserted into the prompt.
- 1-shot with terminology hints: An illustrative example sentence, along with its translation and specific terminology hints, is provided. This example is carefully selected from sentences containing relevant medical terms.
- Embedding-based example retrieval: For the 1-shot approach, examples are not just randomly sampled but retrieved based on semantic similarity using embeddings, aiming to provide more relevant context.
They evaluated six publicly available multilingual LLMs ranging from 0.5 billion to 9 billion parameters.
Datasets
- MedEV dataset: A recently released English-Vietnamese parallel corpus for medical texts, consisting of 360K sentence pairs. It's de-identified and distributed under an open license.
- Meddict: An English-Vietnamese medical dictionary developed at VinUniversity, containing over 64K entries. This was crucial for the dictionary-augmented prompting strategies.
Key Insights and Ideas
- Model Scale is King (Mostly): Larger LLMs generally outperform smaller ones, especially in zero-shot scenarios. This suggests that the inherent knowledge encoded in larger models is a significant advantage for domain-specific translation.
- Few-Shot Isn't Always Better: Simple random few-shot prompting provided only marginal improvements over zero-shot. This implies that for medical translation, simply showing more random examples doesn't necessarily teach the model much new.
- Terminology is Paramount: Explicitly incorporating medical terminology hints, whether in zero-shot or 1-shot settings, consistently improved translation quality across all models. This highlights the critical need for domain-specific knowledge.
- Smart Examples Trump Quantity: Using embedding-based retrieval to select relevant 1-shot examples yielded the most significant and consistent gains, far surpassing random sampling or just adding terminology hints alone. This suggests that providing contextually relevant examples is more effective than just providing more examples.
- Instruction Tuning Matters: Instruction-tuned models generally performed better than SFT-only variants, indicating the benefit of explicit instruction following for translation tasks.
Key Results
- Performance Gains: The Gemma-SEA-LION-v3-9B-IT model achieved the strongest baseline performance, exceeding 36 BLEU on En-Vi translation.
- Dictionary Augmentation Impact: Adding terminology hints improved performance across all models. The most substantial gains came from combining terminology hints with embedding-based example retrieval, leading to improvements of several BLEU points for larger models. For instance, Gemma-SEA-LION-v3-9B-IT saw its BLEU score jump from 37.61 (zero-shot) to 41.71 with this advanced prompting.
- Sentence Length Sensitivity: Translation quality was highest for medium-length sentences (20-50 tokens) and dropped for very short or very long sentences. Larger models were more robust to sentence length variations.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Size Limit: Experiments were capped at 9B parameter models. Larger LLMs might exhibit different behaviors, especially regarding few-shot prompting and terminology handling.
- Automatic Metrics Only: Evaluation relied solely on automatic metrics (BLEU, CometKiwi, XCOMET). Human evaluation, particularly for terminology fidelity and clinical adequacy, would provide stronger evidence.
- Limited Prompting Strategies: The study focused on zero-shot, few-shot, and dictionary-augmented prompting. Other techniques like instruction tuning with domain-specific corpora or reinforcement learning (e.g., DPO) were not explored.
Future work will involve applying fine-tuning techniques (SFT, DPO) and exploring domain-specific adaptation strategies to further enhance medical En-Vi MT. They also plan to conduct human evaluations and investigate terminology-specific subsets for more targeted analysis.
Summary updated