Abstract
The rise of Large Language Models (LLMs) has redefined Machine Translation (MT), enabling context-aware and fluent translations across hundreds of languages and textual domains. Despite their remarkable capabilities, LLMs often exhibit uneven performance across language families and specialized domains. Moreover, recent evidence reveals that these models can encode and amplify different biases present in their training data, posing serious concerns for fairness, especially in low-resource languages. To address these gaps, we introduce Translation Tangles, a unified framework and dataset for evaluating the translation quality and fairness of open-source LLMs. Our approach benchmarks 24 bidirectional language pairs across multiple domains using different metrics. We further propose a hybrid bias detection pipeline that integrates rule-based heuristics, semantic similarity filtering, and LLM-based validation. We also introduce a high-quality, bias-annotated dataset based on human evaluations of 1,439 translation-reference pairs.
Introduction
While LLMs have revolutionized Machine Translation (MT) with their fluency and context-awareness, their performance is far from equitable. Most evaluation metrics (like BLEU or COMET) focus on lexical or semantic accuracy but ignore the critical dimension of representation. This research addresses two primary gaps: the performance gap across linguistic distances and domains, and the bias gap where models encode and amplify harmful stereotypes during translation. Current tools often fail to detect subtle biases that are reflections of imbalanced training data rather than simple linguistic errors.
The Translation Tangles Framework
To expose the friction between translation quality and social fairness, we developed a multi-layered evaluation pipeline. The study utilizes a mix of general (WMT-18, WMT-19, BanglaNMT) and specialized corpora, including legal (MultiEURLEX), medical (ELRC-Medical-V2), and literary (Lit-Corpus) domains.
Our hybrid bias detection pipeline moves beyond simple keyword matching by combining:
- Rule-based heuristics: Utilizing lexicons for gender, religion, and race.
- Named Entity Recognition (NER): Monitoring if models swap or introduce sensitive entities.
- Semantic Similarity Filtering: Ensuring translations maintain the source meaning while checking for bias-driven drift.
- LLM-as-a-Judge: Employing advanced models to validate and explain detected biases.
Key Insights and Results
Our analysis reveals that while scaling model size helps reduce performance gaps between "intra-family" (e.g., French to Spanish) and "cross-family" (e.g., Gujarati to German) translations, it does not eliminate them. Even the largest models, such as Llama-3.2-90b, struggle with difficult pairs like English-Turkish despite achieving top overall scores (BLEU = 44.16).
The Accuracy Paradox
A significant finding is the "Accuracy Paradox": a translation can be grammatically perfect and semantically "correct" according to standard metrics while still being heavily biased. For instance, a model might correctly translate a sentence but swap a neutral term for a gendered stereotype or a Western-centric cultural equivalent.
As shown in the heatmaps, cultural and sociocultural biases are the most frequent, accounting for over 75% of cases. Low-resource languages like Gujarati (gu-en) and Kazakh (kk-en) exhibited significantly higher bias counts than high-resource languages. The framework flagged 1,902 instances of bias in total, with cultural bias being the most prevalent (798 instances).
Domain Sensitivity
Performance is highly domain-dependent. LLMs excel in the Law domain due to its structural regularity. However, Literature remains a significant challenge; BLEU scores dropped by a staggering 68.7% when moving from legal to literary texts, as models fail to capture stylistic nuances and implicit context.
Conclusion and Future Work
While open-source LLMs are increasingly capable, they are often "ready to translate, but not to represent." The current detection pipeline achieved 100% recall against human gold standards but faces challenges with false positives and complex harms like sarcasm or omission bias. Future research will focus on expanding detection to bidirectional translations (English→X) and normalizing for training data volume to better understand the root causes of these representational failures.
Summary updated