TL;DR
This paper introduces a new, high-quality multilingual terminology resource for the tax and financial education domains across seven diverse languages. They use this resource to evaluate various machine translation (MT) systems and large language models (LLMs), revealing significant challenges with specialized terminology and highlighting the benefits of terminology-aware translation.
What Problem Does This Paper Address?
Accurate translation is crucial in high-stakes fields like taxation and finance, where errors can lead to serious misunderstandings and legal issues. While MT systems and LLMs have advanced, they often struggle with low-frequency, domain-specific terms, especially in languages that are typologically distant or less resourced. Existing public datasets for MT research are often limited in language coverage or domain specificity, making it difficult to assess and improve MT performance in these specialized areas. This paper aims to bridge this gap by creating a robust multilingual terminology resource and using it to systematically evaluate MT and LLM performance on domain-specific translation.
Methods
The authors developed a gold-standard multilingual terminology resource by extracting terms from curated U.S. government publications (IRS for tax, CFPB for financial education). They then annotated these terms for domain specificity (Domain-Specific, Domain-Contextual, General) using a combination of LLMs and human review. To evaluate translation performance, they created a parallel corpus from IRS tax-related articles. They then assessed various MT systems (NLLB, MADLAD) and LLMs (Aya-23, Llama-3.1, Mistral, GPT-40) using metrics like chrF++ for overall translation quality and Term Accuracy (TA) for exact term translation. They also investigated terminology-aided translation by incorporating term pairs into prompts and analyzed parallel term extraction capabilities of LLMs.
Datasets
- Newly Introduced Benchmark: A gold-standard multilingual terminology resource for the tax and financial education domains, covering English, Spanish, Russian, Vietnamese, Korean, Chinese (traditional and simplified), and Haitian Creole.
- Parallel Corpus: Extracted from U.S. IRS tax-related articles, covering seven languages.
- Existing Datasets: The authors leveraged terms from the IRS and CFPB glossaries.
Key Insights and Ideas
- Domain-Specific Terms are a Major Hurdle: Even advanced MT systems and LLMs struggle significantly with translating domain-specific terms compared to general or contextual terms, particularly in the tax domain.
- Haitian Creole Remains Challenging: Despite efforts, Haitian Creole consistently presents challenges for translation models, even with specialized resources.
- Terminology-Aided Translation Shows Promise: Incorporating known term pairs into prompts can lead to notable improvements in Term Accuracy, especially for domain-specific terms in the EN→XX direction.
- Term Accuracy is Crucial: Standard metrics like chrF++ don't always capture improvements in term translation, underscoring the need for term-specific evaluation metrics.
- LLMs Excel at Term Extraction: LLMs, particularly GPT-40, demonstrate strong capabilities in extracting exact term translations from parallel text, even for low-resource languages.
Key Results
- Overall Translation Quality: Performance generally declines when translating out of English, with specific languages like Chinese (Traditional) and Haitian Creole being particularly challenging. GPT-40, the closed-source model, generally outperformed others.
- Term Accuracy: Domain-Specific terms consistently showed lower accuracy scores for the IRS dataset compared to Domain-Contextual and General terms. This pattern was less consistent for the CFPB dataset, possibly due to its less specialized nature.
- Terminology-Aided Translation: For EN→XX in the IRS subset, Term Accuracy improved significantly across languages when using terminology-aware prompts.
- Parallel Term Extraction: GPT-40 achieved high accuracy (avg. 0.90+) in extracting exact term translations, outperforming Mistral (avg. 0.66).
Limitations and Future Work
- Data Contamination: The parallel corpus was derived from recently published IRS articles, which might have been included in the pretraining data of large-scale models, potentially inflating performance.
- Limited Linguistic Expertise: Error analysis was constrained by the availability of a Russian-speaking annotator.
- Model Selection: The study used models primarily trained on general-domain data, which might limit their ability to handle specialized contexts.
- Prompt Engineering: The prompt engineering for experiments was limited due to the exploratory nature of the study.
Future work could involve expanding the range of linguistic annotators, fine-tuning models on tax and finance data, exploring a broader array of LLMs, and refining prompt engineering techniques to further enhance MT performance in specialized domains.
Summary updated