Experience Report: Implementing Machine Translation in a Regulated Industry

Marco Zocca, Per Fallgren, David BuffoniEMNLP 2025

TL;DR

This paper details the practical challenges and lessons learned from implementing AI-driven machine translation (MT) within a global medical technology company. They found that no single MT model universally outperforms others, and human validation remains crucial, especially for domain-specific content, highlighting the need for careful tool selection and integration.

What Problem Does This Paper Address?

Companies operating globally, especially in highly regulated industries like medical technology, face a significant challenge in translating vast amounts of content accurately and compliantly across numerous languages. Traditionally, this relies on human translators, but recent advancements in Neural Machine Translation (NMT) and Large Language Models (LLMs) present a compelling alternative. However, the medical industry demands extremely high accuracy and adherence to specific terminology to avoid regulatory issues or adverse patient outcomes. This paper addresses the gap in understanding how to effectively implement and evaluate these new AI translation tools in such a demanding, "low-resource" (due to lack of domain-specific evaluation data) setting.

Methods

The authors implemented a system for evaluating and comparing seven different MT methods, including traditional NMT (Azure Translator), LLMs (GPT-4), and smaller language models (Gemma 12B). They tested these models using three distinct prompting strategies: zero-shot (direct instruction), few-shot (providing a few examples), and refinement-based (using an initial translation and asking for improvements). A key methodological innovation was the use of a custom annotation platform that integrated with cloud storage, ensuring data security and traceability, and allowing human annotators to rank translations from multiple systems for each sentence.

Datasets

The study utilized two internal data sources from Mölnlycke Health Care:

  • Customer Portal Data: 50 sentences from a digital platform supporting healthcare professionals, containing terminology related to medical technology.
  • Web Data: 50 sentences extracted from publicly accessible website content, localized for various markets and specific to healthcare and product promotion.

These sources were used to create a dataset of English sentence pairs translated into five target languages (Czech, French, Norwegian, Italian, Polish). In total, 4,000 translations were generated and ranked by human annotators.

Key Insights and Ideas

  • No Universal Winner: The most striking finding is that no single MT method consistently outperformed others across all language pairs and domains. This suggests that the "best" tool is context-dependent.
  • The "Gold Standard" Isn't Always Best: The pre-validated human translations ("Gold") were not always ranked as the top choice, particularly for domain-specific content. This indicates that advanced AI models can sometimes match or even surpass human translations in perceived quality, especially when domain expertise is less critical.
  • Domain Specificity Matters: The performance difference between "Portal" (domain-specific) and "Web" (general) data highlights the importance of domain adaptation. Reviewer uncertainty increased with more specialized terminology, suggesting that AI models may struggle more with highly technical language without specific fine-tuning.
  • Human Validation is Still Key: Despite AI advancements, human review remains essential for quality assurance and compliance, especially in regulated industries. The paper emphasizes the need for skilled, domain-expert reviewers.
  • UX is Crucial for Annotation: A user-friendly annotation platform significantly reduces cognitive load for validators, improving efficiency and potentially the quality of their judgments.

Key Results

  • An experiment involving over 11,000 ranked translations was conducted.
  • Aggregate human validator rankings showed no single MT method was universally preferred across the five language pairs.
  • For most languages, the "Gold" reference translation was preferred by annotators between 27-42% of the time, but this preference was lower for French.
  • Reviewer uncertainty was higher for domain-specific "Web" data compared to "Portal" data, suggesting AI models face greater challenges with specialized terminology.
  • In some cases, particularly with short sentences, multiple MT alternatives produced identical translations, sometimes matching the "Gold" standard.

Limitations and Future Work

  • Reviewer Coverage: Uneven coverage of reviewers for different language pairs may have biased the rankings.
  • Model Exhaustiveness: The study did not evaluate all possible MT models, particularly those specific to certain language pairs (e.g., OPUS-MT).
  • Static Prompts: The prompts used were static and did not incorporate context-specific information beyond basic translation patterns.
  • Prior Exposure: It cannot be guaranteed that validators had no prior exposure to the material, which could influence their judgments.
  • Future Work: The authors suggest exploring whether LLMs can effectively rank translations themselves, which could offer a more cost-effective validation alternative, though this might introduce new biases. Further research could also focus on controlling for keywords, domain-specific language, and reviewer expertise to better understand performance variations.

Summary updated

BibTeX
@inproceedings{zocca2025experience,
  author = {Marco Zocca and Per Fallgren and David Buffoni},
  title = {{Experience Report: Implementing Machine Translation in a Regulated Industry}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure