The Impact of Domain-Specific Terminology on Machine Translation for Finance in European Languages

Arturo Oncevay, Charese Smiley, Xiaomo LiuNAACL 2025 · DOI

Introduction

Translating financial documents requires more than linguistic fluency; it demands precise navigation of specialized terminology. Terms such as "fiscal stance," "liquidity," and "HICP" carry specific regulatory and economic weights. Mistranslations in this domain can lead to significant consequences, including regulatory non-compliance or flawed investment strategies. Researchers from JPMorgan AI Research have conducted the first comprehensive study on the impact of specialized terminology on machine translation (MT) across 22 European languages, identifying key challenges such as data contamination in open-source datasets and the scarcity of multi-parallel data for lower-resource languages.

Methodology

The study employed a multi-pronged benchmarking approach to evaluate the current state of financial MT:

  • ECB Multi-Parallel Corpus: A "fresh" dataset of 531 segments aligned across 22 languages, scraped from 2024 European Central Bank publications to ensure models had not seen the data during training.
  • Terminology Mapping: A curated glossary of 1,135 English financial terms was mapped to 21 other languages using SimAlign to create a multilingual gold standard.
  • Metrics: The researchers introduced Term Match Accuracy ($T_{acc}$) and analyzed its correlation with standard metrics: chrF++ (lexical overlap) and COMET (semantic meaning).
  • Model Evaluation: Specialized MT models (NLLB-200, MADLAD-400) were compared against large language models (Llama 3, AYA23, TowerInstruct).

Key Insights

Specialization vs. Scale

Task-specific models, notably MADLAD-400 (3B parameters), consistently outperformed larger LLMs (7B-8B parameters) in financial accuracy. MADLAD-400 achieved the highest average COMET score (90.87) and $T_{acc}$ (0.8620), whereas Llama 3 trailed significantly in terminology precision ($T_{acc}$ of 0.6194). This suggests that domain-specific training data is more critical than raw parameter count for niche applications.

The "LLM Terminology Boost"

A pivotal finding is the relationship between terminology and semantic quality. While specialized models generally perform better overall, LLMs like AYA23 exhibit a stronger correlation between terminology accuracy and semantic (COMET) scores. This implies that for LLMs, correctly identifying domain-specific terms is a primary driver of overall translation coherence.

Correlation between terminology accuracy and translation metrics (chrF++ and COMET) for LLMs, showing high alignment (Corr: 0.96 and 0.91).

Linguistic and Morphological Barriers

Languages with complex morphology (e.g., Finnish, Hungarian) or non-Latin scripts (e.g., Bulgarian) present the greatest challenges. These languages often show lower lexical overlap (chrF++) even when semantic accuracy is high. Furthermore, models frequently struggle with idioms, often defaulting to literal translations that alter the intended financial tone—for instance, translating "a leap of faith" into phrases suggesting "impending doom."

Results and Correlation Analysis

The study confirms a strong positive correlation (often > 0.80) between getting terminology right and the overall quality of the translation. However, the strength of this correlation varies by model type. As illustrated in the comparison below, specialized models may achieve higher absolute scores but sometimes show a weaker link between terminology accuracy and semantic quality compared to LLMs, indicating that LLMs rely more heavily on correct terminology to anchor their semantic understanding.

Correlation between terminology accuracy and translation metrics for specialized MT models, showing a lower semantic correlation (Corr: 0.36) despite high lexical alignment.

Conclusion and Future Directions

While specialized MT models currently lead in financial translation, LLMs show significant potential if terminology accuracy can be improved. Future work will explore techniques like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) to provide models with real-time access to glossaries. Additionally, expanding the corpus size and incorporating human evaluation by professional financial translators will be essential for capturing the nuances of fluency and tone that automated metrics may overlook.

Summary updated

BibTeX
@inproceedings{oncevay2025impact,
  author = {Arturo Oncevay and Charese Smiley and Xiaomo Liu},
  title = {{The Impact of Domain-Specific Terminology on Machine Translation for Finance in European Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.140},
}
Figure