TL;DR
This paper investigates the reliability of using large language models (LLMs) as judges for evaluating text generation across multiple languages. They found that even advanced LLMs struggle to provide consistent judgments across languages, particularly in low-resource settings, suggesting they are not yet a fully reliable solution for multilingual evaluation.
What Problem Does This Paper Address?
Evaluating the quality of text generated by AI models is crucial for progress in Natural Language Processing (NLP). Traditionally, this has relied on metrics like BLEU or human annotators. While LLM-as-a-Judge offers a promising, scalable alternative, its effectiveness in multilingual contexts is uncertain. The core problem is whether LLMs can consistently and accurately judge generated text in languages other than English, especially when the same content is presented in different linguistic forms. This is critical because many real-world applications require multilingual evaluation, and human evaluation is expensive and time-consuming.
Methods
The researchers evaluated five different LLMs (including GPT-3.5-turbo, GPT-4o, Gemini-2.0-Flash, Llama-3.3-70b, and Qwen-2.5-72b) across five diverse NLP tasks (Question Answering, Math Question Answering, Summarization, Dialogue Generation, and Machine Translation) using parallel data from 25 languages. They assessed LLM performance using two criteria: a binary "Yes/No" for correctness and a "Grade" from 1-5. The primary metric for consistency was Fleiss' Kappa, which measures inter-rater agreement. They also explored factors influencing consistency, such as language resource availability, task type, prompt design, and model scale.
Datasets
The study utilized five existing datasets that provide parallel data across multiple languages:
- XQuAD: Question Answering
- MGSM: Math Question Answering
- WMT23: Machine Translation
- WikiLingua: Summarization
- XDailyDialog: Dialogue Generation
These datasets were chosen to ensure that the input information remained identical across instances, with language being the only variable, facilitating a direct comparison of LLM judgment consistency across languages.
Key Insights and Ideas
- Multilingual LLM-as-a-Judge is Inconsistent: The most striking finding is that LLMs, even powerful ones like GPT-4o, struggle to maintain consistent judgments across languages. Their average Fleiss' Kappa was around 0.3, indicating poor agreement.
- Low-Resource Languages are a Major Hurdle: Consistency scores were significantly lower for low-resource languages, suggesting that LLMs are less reliable when evaluating text in languages with less representation in their training data.
- Model Scale and Multilingual Training Don't Guarantee Consistency: Contrary to expectations, simply increasing model size or training on multilingual data did not directly improve judgment consistency. This implies that current LLM architectures and training paradigms may not inherently equip them for robust multilingual evaluation.
- Task Matters: The consistency of LLM judgments varied significantly across different NLP tasks. This suggests that LLM evaluators might be more suited for certain types of tasks than others, and their "domain expertise" (or lack thereof) plays a role.
- Grade Judgment is Harder: LLMs showed lower consistency when assigning a numerical grade (1-5) compared to a binary "correct/incorrect" judgment. This indicates that more nuanced evaluation is more challenging for LLMs.
- Prompt Engineering Helps, But Isn't a Silver Bullet: While generating explanations alongside judgments improved consistency, using overly specific rubrics didn't always yield better results, suggesting LLMs might already have implicit knowledge of common tasks.
Key Results
- Fleiss' Kappa Scores are Low: The highest Fleiss' Kappa score achieved was around 0.54 for GPT-4o on the WikiLingua dataset (Yes/No criterion), which is far from the ideal of 1. For the Grade criterion, the highest was 0.32.
- GPT-4o Leads, But Not by Much: GPT-4o generally performed best in terms of consistency, but its accuracy wasn't always superior, suggesting it might apply stricter standards.
- Open-Source Models Show Mixed Results: Qwen-2.5 performed comparably to some closed-source models, but Llama-3.3 showed more limited performance.
- Aya-Expanse's Specialization Didn't Help: The multilingual-specific model Aya-Expanse did not show significant improvements in consistency, even after fine-tuning, suggesting that specialized multilingual training alone isn't sufficient.
- Ensemble Strategy Improves Consistency: The paper proposes an ensemble strategy (majority vote from multiple open-source LLMs) that showed improvement in consistency, offering a practical solution for real-world applications.
Limitations and Future Work
The authors acknowledge several limitations:
- Focus on Pointwise Evaluation: The study primarily used pointwise evaluation due to the difficulty of obtaining parallel multilingual incorrect candidates. Future work could explore pairwise comparison corpora.
- Model Scale Constraints: Due to GPU limitations, they only evaluated models up to approximately 70 billion parameters. Future work should explore larger LLMs.
Promising future directions include developing more robust multilingual LLM-as-a-Judge systems, constructing parallel pairwise corpora for evaluation, and exploring the impact of even larger LLMs.
Summary updated