Large Language Models for Persian-English Idiom Translation

Sara Rezaeimanesh, Faezeh Hosseini, Yadollah YaghoobzadehNAACL 2025 · DOI

Introduction

Idiomatic expressions represent a significant hurdle for machine translation. Because their meanings are figurative rather than literal—for instance, the English phrase "kick the bucket" has no actual relation to buckets—standard Neural Machine Translation (NMT) systems often produce nonsensical results. This challenge is particularly acute for Persian (Farsi), a language rich in cultural metaphors that lacks the extensive digital datasets available for English.

Recent research addresses this gap by introducing PersianIdioms, a comprehensive resource for Persian idiomatic expressions, and evaluating how modern Large Language Models (LLMs) handle these complex linguistic structures compared to traditional NMT systems.

The PersianIdioms Dataset

A primary contribution of this work is the creation of two new datasets designed to benchmark idiomatic translation:

  1. PersianIdioms: A gold-standard collection of approximately 2,200 Persian idioms, including meanings and contextual usage examples for 700 entries.
  2. Parallel Translation Datasets: Two sets of 200 sentences (English → Persian and Persian → English) curated to contain challenging idioms, manually translated and verified by experts.
An example entry from the PersianIdioms dataset, showing the idiom, its meaning in Persian and English, and a contextual example with a gold translation.

Methodology

The researchers evaluated several state-of-the-art models, including closed-source systems like GPT-4o-mini and Claude-3.5-Sonnet, and open-source models such as Qwen-2.5-72B and Command R+. These were compared against traditional NMT models like Google Translate (GT) and NLLB.

Three prompting strategies were tested:

  • SinglePrompt: A direct translation request.
  • Chain-of-Thought (CoT): A multi-step process where the model identifies the idiom and explains it before translating.
  • MultiPrompt: Breaking the CoT steps into independent interactions to reduce model "distraction."

Additionally, a Hybrid Approach was introduced, where an LLM replaces an idiom with a literal explanation in the source language before a dedicated NMT model performs the final translation.

Key Results and Insights

The evaluation revealed several critical findings regarding the current state of idiomatic translation:

Model Performance

Claude-3.5-Sonnet emerged as the leading model across nearly all metrics, demonstrating a superior grasp of Persian cultural nuances. In the Persian → English task, it achieved a top GPT-4o evaluation score of 75.0 using the MultiPrompt setup.

The Hybrid Boost

One of the most significant findings was the effectiveness of the hybrid approach. For smaller or "weaker" LLMs, combining their reasoning capabilities with the fluency of traditional NMT (like Google Translate) significantly closed the performance gap with much larger models.

Knowledge Asymmetry

Models generally performed better translating English idioms into Persian than vice versa. This suggests that LLMs are more familiar with Western cultural metaphors due to the English-centric nature of their training data.

Comprehensive performance comparison of various LLMs and NMT models across different prompting strategies and translation directions.

Conclusion

While LLMs like Claude-3.5-Sonnet are setting new benchmarks for idiomatic translation, traditional NMT systems still struggle with literalism. The success of hybrid approaches suggests a promising path forward: leveraging LLMs for "understanding" and NMT for "fluency." Future work will focus on expanding dataset sizes and exploring specialized fine-tuning to further bridge the gap in cross-cultural linguistic understanding.

Summary updated

BibTeX
@inproceedings{rezaeimanesh2025large,
  author = {Sara Rezaeimanesh and Faezeh Hosseini and Yadollah Yaghoobzadeh},
  title = {{Large Language Models for Persian-English Idiom Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.405},
}
Figure