mR3: Multilingual Rubric-Agnostic Reward Reasoning Models

David Anugraha, Shou-Yi Hung, Zilu Tang, En-Shiun Annie Lee, Derry Tanti Wijaya, Genta Indra Winata

Abstract

Evaluating Large Language Models (LLMs) often relies on "LLM-as-a-judge" frameworks, which frequently struggle with linguistic diversity and lack transparency. We introduce mR3, a massively multilingual reward reasoning model family trained on 72 languages. mR3 provides transparent, rubric-based reasoning traces and scalar scores across various evaluation tasks. Despite its compact size (up to 14B parameters), mR3 achieves state-of-the-art performance on multilingual benchmarks by learning to reason directly in the target language, often outperforming significantly larger teacher models.

Introduction

Current automatic evaluation methods are heavily biased toward English, often failing to capture cultural nuances or logical consistency in low-resource languages like Swahili or Telugu. Furthermore, most reward models act as "black boxes," providing scores without explanations, or providing reasoning only in English. This lack of transparency and flexibility hinders the development of globally aligned AI. mR3 addresses these gaps by offering a rubric-agnostic framework that generates reasoning traces in the same language as the evaluation prompt.

The mR3 Framework

The authors developed a task-agnostic framework that produces three distinct outputs for every evaluation: a reasoning trace (internal thoughts), a concise explanation, and a scalar score.

The mR3 framework supports multilingual inputs and generates reasoning and scores for point-wise, pair-wise, and binary tasks.

Key technical innovations include:

  • Rubric-Agnostic Design: The model can adapt to any provided evaluation rubric, whether human-crafted or LLM-generated, rather than being hard-coded for specific tasks.
  • TGT-TGT Reasoning: The model is trained to "think" and explain in the target language. Research showed that while English is strong for general reasoning, the largest gains occur when the model reasons natively in the target language to capture linguistic nuances.
  • Supervised Fine-Tuning (SFT) with Curriculum Learning: The models were trained using an "Easy-to-Hard" strategy on high-quality reasoning traces, which proved more effective and stable than Reinforcement Learning via Verifiable Reward (RLVR).

Data and Benchmarking

The researchers curated a high-quality 100k sample dataset covering 72 languages, distilled from a larger pool of 3 million examples. The dataset includes diverse tasks such as preference modeling (Human Arena), general knowledge (MMMLU), coding (HumanEval-XL), and safety (PolyGuardMix). A critical step involved discarding "easy" samples that smaller models could already solve, forcing the model to learn complex, multi-step reasoning.

Experimental Results

mR3 models were evaluated against a suite of multilingual benchmarks, including m-RewardBench, MM-Eval, and IndoPref.

Performance comparison of mR3 models against base models and existing reward models across multilingual benchmarks.

The results demonstrate that specialized reasoning data can compensate for model size:

  • State-of-the-Art Accuracy: mR3-Qwen3-14B achieved an average accuracy of 85.04% across benchmarks, surpassing the 120B-parameter GPT-OSS teacher model and the 49B-parameter Nemotron-Multilingual.
  • Zero-Shot Generalization: The model exhibited high-quality reasoning even for languages not seen during training, suggesting a cross-lingual transfer of reasoning capabilities.
  • Human Alignment: Native speakers across 12 languages consistently preferred mR3’s reasoning traces over base models, noting superior cultural alignment and fluency.

Conclusion and Future Work

mR3 demonstrates that "thinking" natively in a target language is essential for fair and transparent AI evaluation. By providing a rubric-agnostic, multilingual reasoning model, this work paves the way for more inclusive LLM development. Future research will explore iterative rubric refinement, where models critique and improve their own evaluation criteria, and expanding the framework to a wider variety of model architectures.

Paper Citation: Anugraha, D., et al. (2026). mR3: Multilingual Rubric-Agnostic Reward Reasoning Models. Published as a conference paper at ICLR 2026.

Summary updated

BibTeX
@inproceedings{anugraha2026mr,
  author = {David Anugraha and Shou-Yi Hung and Zilu Tang and En-Shiun Annie Lee and Derry Tanti Wijaya and Genta Indra Winata},
  title = {{mR3: Multilingual Rubric-Agnostic Reward Reasoning Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=ST0wOB1bdX},
}
Figure