In the world of AI deployment, we often treat English performance as a proxy for global capability. We pick the "best" model on an English leaderboard, set a confidence threshold based on English validation data, and ship it globally.
A new paper, XDecideBench, suggests this "English-first" strategy is a dangerous gamble. By introducing a massive benchmark for Typed Decision Models (TDMs)—models designed to choose from a fixed set of options rather than generate free-form text—the authors reveal that global model rankings are far more volatile than we realized.
TL;DR
XDecideBench is a comprehensive new benchmark evaluating Typed Decision Models across 122 languages and 9 task suites. It demonstrates that the "best" model in English often fails significantly in other languages and that confidence policies calibrated on English data fail to maintain their error targets when applied globally.
What Problem Does This Paper Address?
Most real-world AI applications aren't just chatbots; they are decision engines. They route customer requests to specific departments, flag abusive content, or rate the sentiment of a message on a fixed scale. These are "typed decisions."
Current benchmarks mostly focus on English or generative tasks. Developers often assume that if a model is top-tier in English, it will remain top-tier in Swahili or Burmese. Furthermore, to ensure safety, developers often set a confidence threshold (e.g., "only act if the model is 95% sure"). The authors wanted to know: Do these rankings and thresholds actually hold up across 122 different languages?
Methods
The authors define TDMs as models that return probabilities for a declared set of answers (CHOICE, YES/NO, or an ordered SCORE). They evaluate these models using a common scoring contract based on the Brier Skill Score (BSS), which measures the quality of predicted probabilities rather than just raw accuracy.
They compare two types of models:
- Decision Heads: Models specifically fine-tuned to return probabilities for fixed options (e.g., Microsoft’s Decision-1 or Cloudflare’s Clef).
- Backbone Readouts: Untuned base models (like Qwen) where probabilities are extracted via log-likelihoods.
The study also tests Confidence Policy Transfer: if you set a threshold in English to allow only a 5% error rate, what is the actual error rate when you apply that same threshold to German or Urdu?
Datasets
XDecideBench is a massive collection of 88,425 texts and 153,875 questions across 122 languages. It categorizes data into three regimes:
- Aligned: Translated versions of the same items (e.g., Belebele, XNLI).
- Locally Labelled: Items where local annotators might disagree with English ones (e.g., RTP-LX toxicity).
- Native: Items originally written by native speakers in their own language (e.g., INCLUDE exams).
Key Insights and Ideas
- The English Mirage: A model that is "confidently right" in English can be "confidently wrong" in another language. For example, Microsoft’s Decision-1 is the top performer on English Belebele but falls to 7th place on Burmese Belebele.
- The "Head" Advantage in Low-Resource Settings: Fine-tuning a "decision head" on top of a backbone model provides a massive boost. Interestingly, the accuracy gain is much larger in low-web-share (low-resource) languages. This isn't because the fine-tuning is "smarter" there, but because the base backbone models are significantly weaker in those languages.
- Rankings are Task-Dependent: A model’s rank changes more based on the suite (the type of task) than the language. A model that wins at routing might lose at toxicity detection.
- Confidence is Not Universal: Confidence thresholds are not portable. A model's "95% certain" in English might only be "80% certain" in reality when processing French or German.
Key Results
- Ranking Volatility: The model that wins in English loses significantly in about 25% of the suite-language combinations tested.
- Burmese Failure: On Burmese Belebele, the English-best model achieved a Brier skill of only .336, while the actual best model for that language (Jev) reached .814.
- Head vs. Backbone: Fine-tuned heads removed between 30% and 44% of their backbones' errors across all tested languages.
- Threshold Drift: A threshold set on English data to allow a 5% error rate resulted in actual error rates of 7.8% to 12.4% on German and French XNLI. In low-resource languages, the error rate at that same threshold spiked even higher.
Limitations and Future Work
- Data Contamination: Since many of the source datasets (like XNLI or Belebele) are public, they may have been included in the training data of the models being tested.
- English-Centric Schemas: Most tests used English instructions and option names (e.g., "A flower"), even for non-English text. The authors found that using native-language instructions actually lowered performance for some models, suggesting they are heavily optimized for English prompts.
- Web Share as a Proxy: "Web share" is a rough proxy for pretraining exposure and is often confounded with script types and language families.
- Future Work: The authors call for more native-authored datasets and emphasize that practitioners must fit their confidence policies on target-language data rather than relying on English transfers.
Summary updated