Calibration and Conformal Prediction for Medical Question Answering in Arabic and English: Evidence from Two Non-Parallel Benchmarks

MariyaarXiv 2026 · arXiv

TL;DR

Accuracy alone is a dangerous metric for medical AI; we need to know when a model’s "confidence" actually correlates with being right. This paper evaluates two open-weight LLMs (Qwen and Jais) on English and Arabic medical exams, demonstrating that while simple techniques like temperature scaling can fix overconfidence, the resulting "calibrated" models are often only useful if their underlying accuracy is already high.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly benchmarked on medical licensing exams, a critical gap has emerged: reliability. In a clinical setting, a model that is confidently wrong is far more dangerous than one that admits it doesn't know the answer.

This paper tackles the challenge of Uncertainty Quantification (UQ). Specifically, it asks:

  1. How well does a model’s predicted probability match its actual accuracy (Calibration)?
  2. Can we generate a set of possible answers that is guaranteed to contain the correct one (Conformal Prediction)?
  3. Can we reduce errors by only answering questions where the model is highly confident (Selective Prediction)?

Crucially, the author explores these questions across two languages—English and Arabic—using models of different lineages to see if these UQ techniques hold up in a cross-lingual context.

Methods

The study evaluates two 7B-parameter models: Qwen2.5-7B-Instruct and Jais-adapted-7B-chat (an Arabic-centric model), both run in 4-bit precision to reflect realistic hardware constraints. The author employs three main technical pillars:

  • Temperature Scaling: A post-hoc calibration method where a single parameter ($T$) is used to "soften" or "sharpen" the model's probability distribution. This helps correct the common tendency of neural networks to be overconfident.
  • Split Conformal Prediction: This framework turns raw model scores into a prediction set. Instead of picking one answer, the model provides a list of options that is mathematically guaranteed to contain the correct answer at a user-defined confidence level (e.g., 90%).
  • Selective Prediction: The author ranks questions by the "margin" (the difference between the top two option probabilities) and evaluates the error rate when the model only answers the most confident $X\%$ of questions.

Datasets

The evaluation uses two distinct, non-parallel benchmarks:

  • MedMCQA (English): 2,816 single-choice questions from Indian medical entrance exams.
  • MedAraBench (Arabic): 4,927 questions across 19 specialties.

The author notes that because these datasets differ in subject matter and difficulty, direct comparisons between English and Arabic performance should be treated as confounded by the data itself, rather than just the language.

Key Insights and Ideas

  • Calibration $\neq$ Usefulness: This is perhaps the most important "aha" moment. Temperature scaling successfully reduced the Expected Calibration Error (ECE) by over 80% for all models. However, for the Jais model on Arabic, the "calibrated" confidence was often just a flat line near the chance level (around 25-30%). The model "knew" it was guessing, which makes it calibrated, but not particularly helpful for a user.
  • The "C" Bias: The Jais model showed a massive prior bias toward option "C" (predicting it 60% of the time in Arabic). Interestingly, scoring the model based on Latin letters vs. Arabic script spellings of the options significantly shifted this bias, though it didn't drastically change overall accuracy.
  • Conformal Sets as a Difficulty Heatmap: Conformal prediction reached its target coverage perfectly. However, when accuracy was low, the "90% confidence set" often included almost every available option. This effectively signals to the user: "I have no idea; it could be anything."
  • The Power of Silence: For the Qwen model on English, answering only the top 20% most confident questions slashed the error rate from 43.9% to just 9.8%. This suggests that for high-performing models, selective prediction is a viable path toward "safe" AI.

Key Results

The quantitative results highlight a clear hierarchy in performance and reliability:

Table 2. Accuracy and AUROC results.
  • Accuracy: Qwen outperformed Jais significantly (0.561 vs 0.369 in English; 0.430 vs 0.308 in Arabic).
  • Calibration Fix: As shown in Figure 1, raw models (blue lines) were wildly overconfident, staying far below the ideal diagonal. Temperature scaling (orange lines) brought them much closer to the diagonal.
Figure 1. Reliability diagrams, raw and temperature scaled, for each model and language (one random 50/50 split; the temperatures in the legends come from that split and differ slightly from the 20-split means in Table 3). Questions per bin for the scaled curves are given in Table B1.
  • ECE Reduction: Raw calibration error ranged from 0.242 to 0.486. After scaling, it dropped to between 0.035 and 0.068.
  • Conformal Informativeness: At 90% target coverage, Qwen’s English sets contained 63% of options on average, while Jais’s Arabic sets were nearly uninformative, containing 89% of all options.

Limitations and Future Work

The author identifies several key constraints:

  • Non-Parallel Data: The primary limitation is that the English and Arabic benchmarks are not translations of each other. Future work should use parallel datasets to isolate the effect of language on calibration.
  • Model Scale: The study was limited to 7B models. Larger models (e.g., 70B+) often exhibit better "self-knowledge" and might show different calibration patterns.
  • Prompt Sensitivity: Only one prompt format was used per model; LLM confidence is notoriously sensitive to how a question is phrased.
  • Clinical Reality: These are exam questions, not clinical cases. The author explicitly warns that these results should not be taken as evidence of safety for actual patient care.

For those interested in the implementation, the author has made the code available on GitHub.

Summary updated

Abstract

Large language models (LLMs) are increasingly tested on medical exam questions, but accuracy alone does not show when an answer can be trusted. I study the calibration and uncertainty of two open-weight models, Qwen2.5-7B-Instruct and Jais-adapted-7B-chat (both run in 4-bit precision), on 2,816 English multiple-choice questions from MedMCQA and 4,927 Arabic multiple-choice questions from MedAraBench. Accuracy was 0.561 (English) and 0.430 (Arabic) for Qwen, and 0.369 and 0.308 for Jais. Raw option probabilities were overconfident in all four settings (expected calibration error 0.242 to 0.486), and a single temperature parameter removed 82% to 93% of that error. Split conformal prediction reached its nominal coverage in all 24 settings I tested (within 0.006), but the sets were large when accuracy was low: at 90% coverage, temperature-scaled sets contained on average 63% (Qwen, English), 78% (Qwen, Arabic), 86% (Jais, English) and 89% (Jais, Arabic) of the options. The margin between the top two option probabilities separated correct from incorrect answers best for Qwen on English (AUROC 0.734) and worst for Jais on Arabic (0.579). Answering only the 20% most confident questions lowered Qwen's English error from 0.439 to 0.098. Jais showed a strong prior for option C (49% of English and 60% of Arabic predictions). Scoring Latin letters only cut the Arabic share to 29% but left accuracy, raw ECE and AUROC nearly unchanged. Because the two benchmarks differ in subject, level and number of options, I do not attribute differences between languages to language itself. Code is available at https://github.com/Mariyaaleem/arabic-english-medqa-uncertainty.
BibTeX
@misc{mariya2026calibration,
  author = {Mariya},
  title = {{Calibration and Conformal Prediction for Medical Question Answering in Arabic and English: Evidence from Two Non-Parallel Benchmarks}},
  year = {2026},
  eprint = {2611.16821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16821},
}
Figure