TL;DR
This paper investigates the surprisingly poor performance of large language models (LLMs) in translating numerical data, particularly large units like "million" and "billion," between Chinese and English. The authors introduce a new dataset and demonstrate that even state-of-the-art LLMs struggle with these tasks, often making significant errors. They propose and evaluate three strategies to improve numerical translation, finding that post-editing offers the most promising results.
What Problem Does This Paper Address?
Accurate translation of numerical data is critical across many domains, from finance to medicine. Errors in numerical translation can have severe consequences, leading to financial losses or misinformation. While LLMs have shown remarkable progress in general natural language processing and translation, their ability to handle numerical data, especially complex unit conversions and precise figures, remains largely unexplored. Existing evaluation metrics like BLEU and COMET are also insufficient for capturing subtle but critical numerical translation errors. This paper aims to systematically evaluate LLMs' numerical translation capabilities and identify strategies to mitigate their shortcomings in this area.
Methods
The authors constructed a novel, bi-directional Chinese-English numerical translation dataset based on real-world business data, encompassing ten distinct types of numerical translation. They then evaluated several prominent open-source LLMs (including Llama3.1 series, Tower-7B, and Qwen) and a commercial tool (Google Translate) on this dataset. To address the identified weaknesses, they proposed and investigated three strategies for improving large unit translation:
- In-context Learning (ICL): Providing LLMs with specific unit conversion principles within the prompt.
- Chain-of-Thought (COT): Prompting LLMs to break down the translation task into steps, focusing on numerical parts first.
- Post-Editing (PE): Using LLMs to extract numerical translation pairs from source-target sentences, then employing external tools (cn2an, en2an) to convert numbers to digits for verification and correction.
Datasets
The primary dataset introduced in this paper is a newly constructed bi-directional Chinese-English numerical translation dataset. This dataset is derived from real business data and covers ten specific types of numerical translation: Large unit, Range, Decimal, Number String, Fraction, Ratio, Negative Numbers, Formula, Ordinal, and Special. The authors note that the distribution of these types reflects real-world data.
Key Insights and Ideas
- LLMs struggle with numerical precision: Despite their advanced language capabilities, LLMs exhibit a surprising lack of precision when dealing with numerical data, especially large units and complex calculations. This suggests their underlying architecture, while excellent for sequential language patterns, is not inherently suited for precise mathematical operations or unit conversions.
- Large units are a major bottleneck: The translation of large numerical units (e.g., "million," "billion," "亿") is a particularly weak point for LLMs, with error rates as high as 20% even for advanced models.
- Existing evaluation metrics are insufficient: Standard MT evaluation metrics fail to adequately penalize or even detect critical numerical translation errors.
- Post-editing offers a robust solution: While ICL and COT show some improvement, the Post-Editing (PE) strategy, which leverages LLMs for extraction and external tools for correction, significantly outperforms other methods for large unit translation. This highlights the effectiveness of a hybrid approach that combines LLM strengths with specialized tools.
Key Results
- Widespread errors: Experiments showed that most open-source LLMs faltered when translating numerical data, with significant error rates observed across various types.
- Large unit translation is particularly problematic: Even the latest Llama3.1 8b model showed error rates as high as 20% for numerical types involving large units.
- GLM-4-9b-chat as a top performer (but still imperfect): This model achieved the highest average accuracy (97.2% EN-ZH, 96.2% ZH-EN) but still struggled with large units, averaging 81.9% and 78% respectively.
- PE strategy's effectiveness: In Table V, the PE strategy consistently outperformed ICL and COT across all tested base models (BLOOMZ-7B-MT, Mistral-7B, Llama3.1-8B) for large unit translation, achieving pass rates as high as 95.6% (EN-ZH) and 96.3% (ZH-EN).
Limitations and Future Work
- LLM architecture limitations: The authors posit that LLMs' Transformer architecture is better suited for sequential data and language patterns than for precise numerical computations and unit conversions.
- Precision issues in floating-point calculations: LLMs face inherent precision challenges in numerical operations, leading to accumulated errors in complex calculations.
- Future research directions: The paper suggests that future research should continue to explore LLM-based numerical translation and develop more effective strategies. The authors' work on post-editing provides a promising avenue for improving the reliability of LLMs in this critical domain.
Summary updated