Is Jev a Good Multilingual Judge?

Wei Liu, Yi Fan, Xiyan FuarXiv 2026 · arXiv

TL;DR

Jev, a specialized decision model, offers a highly cost-effective alternative to generative LLM judges for multilingual evaluation, operating at roughly 1/30th to 1/60th the cost of GPT-6.1. While it is competitive on many tasks and provides reliable internal confidence scores for error detection, it struggles with consistency in low-resource languages, where it tends to be overly lenient toward incorrect answers.

What Problem Does This Paper Address?

As large language models (LLMs) are deployed globally, we need automated "judges" to evaluate their outputs across dozens of languages. Currently, researchers typically use massive generative models like GPT-4 (or in this paper, the hypothetical GPT-6.1) as judges. However, these models are computationally expensive and slow.

Specialized "decision models" like Jev offer a potential solution. Unlike generative models that write out a justification, decision models directly output probabilities for specific judgments (e.g., "Is this answer correct? Yes/No"). While Jev has shown promise in English, its reliability as a multilingual judge remained largely unstudied. This paper investigates whether Jev can maintain accuracy and consistency across diverse languages and evaluation formats (binary, pairwise, and graded).

Methods

The authors conducted a systematic head-to-head comparison between Jev (v1.13.0) and two generative benchmarks: Qwen3.8-27B and GPT-6.1.

Key methodological choices included:

  • Evaluation Formats: They tested binary decisions (Yes/No), pairwise comparisons (Which is better?), and graded ratings (1–5 scale).
  • Instruction Language: They tested whether writing the evaluation prompt in English or the target language affected performance.
  • Confidence Analysis: Since Jev outputs probabilities, the authors calculated AUROC to see if Jev’s "confidence" in a judgment correlated with that judgment actually being correct.
  • Model Routing: They explored a "cascade" strategy—using Jev for most tasks but routing low-confidence samples to a more expensive model (GPT-6.1).

Datasets

The study utilized four diverse multilingual benchmarks covering up to 47 language varieties:

  1. MEMERAG: Evaluates RAG (Retrieval-Augmented Generation) faithfulness at the sentence level (5 languages).
  2. Global-MMLU-Lite: Answer verification for multiple-choice questions (23 languages).
  3. MENLO: Assessing response quality based on cultural relevance and general helpfulness (47 languages).
  4. METAL: Summary quality evaluation, including hallucination detection (10 languages).

Key Insights and Ideas

  • The "False Acceptance" Bias: In low-resource languages (like Yoruba or Welsh), Jev’s performance drop is primarily driven by "false acceptances." It often accepts incorrect answers in these languages that it correctly rejects when presented in English.
  • Consistency vs. Accuracy: While Jev is more consistent across languages than Qwen, it trails GPT-6.1. Interestingly, GPT-6.1 maintains incredibly high consistency even when its absolute accuracy varies.
Figure 1: Agreement with English judgments across 22 non-English languages in Global-MMLU-Lite. Outer bands indicate high-, mid-, and low-resource language groups defined in the dataset (Singh et al., 2025).
  • English is the Universal Language for Rubrics: Translating evaluation instructions into the target language provided no performance gain but significantly increased token costs. The authors recommend using English instructions by default for multilingual judging.
  • Probabilities are Actionable: Jev’s probability outputs are a strong signal for reliability, particularly in binary tasks. If the model is "unsure," it is much more likely to be wrong.
Figure 4: How well Jev’s confidence distinguishes cor- rect from incorrect judgments, measured by AUROC. Colors distinguish judgment types, and the dashed line marks chance level (0.5).

Key Results

  • Massive Cost Savings: Jev costs between $0.026 and $0.082 per 1,000 samples, whereas GPT-6.1 costs $0.82 to $5.01. On average, GPT-6.1 is ~37x more expensive.
  • Competitive Performance: Jev achieved the highest balanced accuracy on METAL hallucination detection (85.9%) and the highest macro-F1 on MENLO ratings (33.3%), outperforming both Qwen and GPT-6.1 in these specific areas.
  • Knowledge Gap: Jev falls behind on knowledge-intensive tasks. On Global-MMLU-Lite answer verification, it scored 89.6% BAcc, significantly lower than GPT-6.1’s 96.1%.
  • Effective Routing: By routing the bottom 10% of Jev's lowest-confidence samples to GPT-6.1, the authors improved Global-MMLU-Lite performance from 89.57% to 92.73%, demonstrating a viable path for balancing cost and quality.
Figure 2: Breakdown of judgment changes from English to other languages.

Limitations and Future Work

  • Low-Resource Leniency: Jev’s tendency to be "too nice" to incorrect outputs in low-resource languages suggests a need for better multilingual pre-training or calibration for decision models.
  • Graded Ratings: While Jev is excellent at binary and pairwise tasks, its confidence signal is much weaker for graded ratings (1-5 scales), making it harder to detect errors in that format.
  • Knowledge Intensity: The model's struggle with fact-heavy verification suggests it may need to be paired with external knowledge retrieval to compete with the largest LLMs on those specific tasks.

Summary updated

Abstract

Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately $1/31$ to $1/61$ of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.
BibTeX
@misc{liu2026is,
  author = {Wei Liu and Yi Fan and Xiyan Fu},
  title = {{Is Jev a Good Multilingual Judge?}},
  year = {2026},
  eprint = {2611.14861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14861},
}
Figure