XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages

Marek Šuppa, Jaroslav KopčanarXiv 2026 · arXiv

In the world of AI deployment, we often treat English performance as a proxy for global capability. We pick the "best" model on an English leaderboard, set a confidence threshold based on English validation data, and ship it globally.

A new paper, XDecideBench, suggests this "English-first" strategy is a dangerous gamble. By introducing a massive benchmark for Typed Decision Models (TDMs)—models designed to choose from a fixed set of options rather than generate free-form text—the authors reveal that global model rankings are far more volatile than we realized.

TL;DR

XDecideBench is a comprehensive new benchmark evaluating Typed Decision Models across 122 languages and 9 task suites. It demonstrates that the "best" model in English often fails significantly in other languages and that confidence policies calibrated on English data fail to maintain their error targets when applied globally.

What Problem Does This Paper Address?

Most real-world AI applications aren't just chatbots; they are decision engines. They route customer requests to specific departments, flag abusive content, or rate the sentiment of a message on a fixed scale. These are "typed decisions."

Current benchmarks mostly focus on English or generative tasks. Developers often assume that if a model is top-tier in English, it will remain top-tier in Swahili or Burmese. Furthermore, to ensure safety, developers often set a confidence threshold (e.g., "only act if the model is 95% sure"). The authors wanted to know: Do these rankings and thresholds actually hold up across 122 different languages?

Methods

The authors define TDMs as models that return probabilities for a declared set of answers (CHOICE, YES/NO, or an ordered SCORE). They evaluate these models using a common scoring contract based on the Brier Skill Score (BSS), which measures the quality of predicted probabilities rather than just raw accuracy.

They compare two types of models:

  1. Decision Heads: Models specifically fine-tuned to return probabilities for fixed options (e.g., Microsoft’s Decision-1 or Cloudflare’s Clef).
  2. Backbone Readouts: Untuned base models (like Qwen) where probabilities are extracted via log-likelihoods.

The study also tests Confidence Policy Transfer: if you set a threshold in English to allow only a 5% error rate, what is the actual error rate when you apply that same threshold to German or Urdu?

Datasets

XDecideBench is a massive collection of 88,425 texts and 153,875 questions across 122 languages. It categorizes data into three regimes:

  • Aligned: Translated versions of the same items (e.g., Belebele, XNLI).
  • Locally Labelled: Items where local annotators might disagree with English ones (e.g., RTP-LX toxicity).
  • Native: Items originally written by native speakers in their own language (e.g., INCLUDE exams).
Overview of XDecideBench suites and language coverage

Key Insights and Ideas

  • The English Mirage: A model that is "confidently right" in English can be "confidently wrong" in another language. For example, Microsoft’s Decision-1 is the top performer on English Belebele but falls to 7th place on Burmese Belebele.
  • The "Head" Advantage in Low-Resource Settings: Fine-tuning a "decision head" on top of a backbone model provides a massive boost. Interestingly, the accuracy gain is much larger in low-web-share (low-resource) languages. This isn't because the fine-tuning is "smarter" there, but because the base backbone models are significantly weaker in those languages.
  • Rankings are Task-Dependent: A model’s rank changes more based on the suite (the type of task) than the language. A model that wins at routing might lose at toxicity detection.
  • Confidence is Not Universal: Confidence thresholds are not portable. A model's "95% certain" in English might only be "80% certain" in reality when processing French or German.
Figure 1: The best model in English is confidently wrong on this Burmese item. A real Belebele item, selected after the runs, with the Burmese options glossed in English. Bars show the probabilities of Microsoft’s Decision-1, the best model on English Belebele. Two options that both Decision-1 and Jev put near zero are omitted. Over all Burmese items its Brier skill is .336, against .814 for Jev (§6.1).

Key Results

  • Ranking Volatility: The model that wins in English loses significantly in about 25% of the suite-language combinations tested.
  • Burmese Failure: On Burmese Belebele, the English-best model achieved a Brier skill of only .336, while the actual best model for that language (Jev) reached .814.
  • Head vs. Backbone: Fine-tuned heads removed between 30% and 44% of their backbones' errors across all tested languages.
  • Threshold Drift: A threshold set on English data to allow a 5% error rate resulted in actual error rates of 7.8% to 12.4% on German and French XNLI. In low-resource languages, the error rate at that same threshold spiked even higher.
Figure 4: Accuracy gains grow as web share falls. Heads against their own backbones per Core12 lan- guage, by decreasing web share. (a) Accuracy gain. (b) The gain as a share of the backbone’s errors, which barely changes under the local readout. Solid: local readout. Pale, dotted: hosted readout. Bars: 95% paired intervals. Segments: group means. Shaded: low-web- share group.

Limitations and Future Work

  • Data Contamination: Since many of the source datasets (like XNLI or Belebele) are public, they may have been included in the training data of the models being tested.
  • English-Centric Schemas: Most tests used English instructions and option names (e.g., "A flower"), even for non-English text. The authors found that using native-language instructions actually lowered performance for some models, suggesting they are heavily optimized for English prompts.
  • Web Share as a Proxy: "Web share" is a rough proxy for pretraining exposure and is often confounded with script types and language families.
  • Future Work: The authors call for more native-authored datasets and emphasize that practitioners must fit their confidence policies on target-language data rather than relying on English transfers.

Summary updated

Abstract

Applications use Typed Decision Models to route and moderate text in many languages, and they often act only on decisions above a confidence threshold. We introduce XDecideBench, a benchmark of nine suites in 122 languages with 88,425 texts and 153,875 typed questions. All suites share one scoring contract, and we report aligned, locally labelled and native items separately. Model rankings change from suite to suite. The model that is best in English, chosen in hindsight, loses significantly in about a quarter of the suite-language cells, and on Burmese Belebele it falls far behind the best model. Three open models fine-tuned for typed decisions beat the Qwen models they were built from in all twelve development languages. When the Qwen models are run locally, a pre-registered held-out cohort confirms that the accuracy gain is larger in the four languages with the least web text than in the six with the most. In both sets of languages, however, fine-tuning fixes a similar share of the base model's errors, so the larger gain may reflect weaker base models. A confidence threshold set on English data to allow 5% errors exceeds that error rate on XNLI, even in German and French.
BibTeX
@misc{suppa2026xdecidebench,
  author = {Marek Šuppa and Jaroslav Kopčan},
  title = {{XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages}},
  year = {2026},
  eprint = {2611.01370},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01370},
}
Figure