Abstract
While modern Large Language Models (LLMs) excel at translating standard narrative text, they struggle significantly with "edge case" genres like poetry, legal documents, and casual conversation. This paper introduces the Complex Structures Test (CoST) suite, a curated dataset of 1,947 English sentences designed to evaluate English-to-Hindi translation across nine distinct categories. By evaluating 16 state-of-the-art systems, we demonstrate that high average performance scores often mask critical failures in handling stylistic nuances and archaic linguistic structures.
Introduction
Traditional evaluation of Neural Machine Translation (NMT) often relies on random text samples and aggregate metrics like BLEU. While these provide a general sense of quality, they frequently obscure subtle but vital failures. A model might achieve a high score on news articles but fail completely when faced with the metaphorical language of poetry, the rigid formality of a legal contract, or the idiomatic flow of a screenplay. The authors address this gap by creating a "challenge set" specifically designed to "break" LLMs, moving beyond generic evaluations to pinpoint exactly which linguistic structures and domains remain "unsolved" for English-Hindi translation.
Methods
The researchers employed a multi-faceted evaluation strategy using the CoST Suite, a curated dataset of 1,947 English sentences selected for their structural complexity (sentence length, dependency tree depth, and complex phrase combinations). The suite draws from 16 sources across 9 distinct categories, including Poetry (Kabir, Amir Khusro), Narration (Tagore, Dickens), Legal (court judgments), Technical (academic papers), and others like Plays and Conversation.
Evaluation involved both reference-less metrics (COMET-Kiwi) and reference-based metrics (BLEU, chrF, MEE4, BERTScore, and COMET) against professional human translations. Additionally, professional native Hindi speakers performed qualitative reviews to identify specific error types such as hallucinations and grammatical blunders.
Key Results
System Performance and the Narrative Bias
Models show a clear performance split, excelling at narrative styles (autobiography, prose) likely due to their dominance in training data. However, they underperform significantly in poetry, legal, and conversational data. TranssionMT emerged as the overall winner, consistently ranking #1 across nearly all metrics, followed closely by Claude-3.5 and ONLINE-B.
There is a massive disparity between top-tier and bottom-tier models. For instance, while TranssionMT achieved a BLEU score of 68.399, the CycleL system trailed at a nearly unusable 1.77. Interestingly, both reference-free and reference-based metrics generally agreed on system rankings.
Qualitative Insights
- Contextual Over-thinking: The
ONLINE-Gmodel translated the name "Phoebe" as "चाँद" (Moon) because Phoebe is a moon of Saturn, prioritizing astronomical knowledge over simple name transliteration. - The Nasalization Trap: Almost all models (except Llama3-70B and IOL_Research) consistently failed a basic Hindi spelling rule regarding nasalization marks in the word for "am".
- Archaic vs. Modern: The archaic styles found in legal and technical writing proved to be a major hurdle, suggesting that LLMs struggle with formal linguistic structures that deviate from modern, web-scraped English.
Conclusion
The sub-par performance of LLMs in creative and formal domains highlights a need for more diverse training data. Future work should focus on domain-specific refinement for tone and lexical choice, reducing hallucinations in named entity recognition, and addressing recurring errors in Hindi-specific grammar. The CoST test suite is publicly available to support further research in this area.
Summary updated