A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

Boyang Xue, Hongru Wang, Weichao Wang, Rui Wang, Sheng Wang, Zeming Liu, Kam-Fai Wong

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, yet their tendency to "hallucinate"—generating factually incorrect information with high confidence—remains a significant barrier to deployment in high-stakes environments. While confidence estimation (CE) has emerged as a vital technique for mitigating these risks, existing research has been predominantly confined to English. This creates a critical gap in our understanding of AI reliability: does a model’s ability to gauge its own certainty hold steady across different languages? This paper introduces MLINGCONF, a comprehensive benchmark designed to evaluate multilingual confidence estimation, revealing that linguistic dominance significantly impacts a model's calibration.

The Problem: Linguistic Dominance

The researchers hypothesize that because LLMs are trained on imbalanced datasets, their internal representation of certainty is tied to the volume and quality of data in a specific language. To test this, they distinguish between two types of tasks:

  1. Language-Agnostic (LA) tasks: These involve general reasoning (math, science, logic) where the underlying truth is independent of the language used.
  2. Language-Specific (LS) tasks: These involve questions deeply rooted in specific cultural, geographical, or historical contexts (e.g., Japanese regional traditions).

Methodology

The study evaluates three distinct approaches to confidence estimation:

  • Probability-based (Prob.): A "white-box" approach calculating the geometric mean of token-level probabilities from the model's internal logits.
  • p(True)-based: A prompting method where the model is asked "Is your answer true?" and the probability of the token "True" is extracted.
  • Self-verbalized (Verb.): A direct method where the model is instructed to provide a numerical confidence score (e.g., 0.8) within its natural language response.

A key contribution of this work is the introduction of Native-Tone Prompting (NTP). Recognizing that models often perform better when the prompt language matches the cultural domain of the question, NTP involves a two-step process: first, identifying the linguistic/cultural context of the query, and second, using that specific language for both answer generation and confidence estimation.

The MLINGCONF Dataset

The authors constructed a robust evaluation suite covering five diverse languages: English (en), Chinese (zh), Japanese (ja), French (fr), and Thai (th).

  • LA Tasks: The researchers translated four standard English benchmarks—TriviaQA (facts), GSM8K (math), CSQA (common sense), and SciQ (science)—into the target languages using GPT-4, followed by rigorous human expert verification.
  • LS Task (LSQA): They developed a new Language-Specific QA dataset targeting information where a particular language holds "linguistic dominance," such as French history or Thai geography.

Key Insights and Results

  • English Hegemony in General Logic: In general reasoning tasks (LA), English remains the dominant language. Models are not only more accurate when processing English but also exhibit lower Expected Calibration Error (ECE), meaning their confidence scores more accurately reflect their actual performance.
  • The Native Advantage in Culture: For LS tasks, the trend shifts. Models demonstrate significantly better calibration when prompted in the language associated with the topic. The NTP strategy achieved an accuracy of 79.46% and an ECE of 10.28, outperforming English-only prompting (ECE of 12.15).
  • Model-Specific Strengths: The effectiveness of CE methods varies by model capability. High-performing models like GPT-3.5 excel with prompt-based methods (p(True) or Verbalized), while models with weaker instruction-following, such as Llama-3.1, are more stable using raw token probabilities.
  • Math as a Universal Language: Interestingly, mathematical reasoning (GSM8K) showed the least amount of language bias, suggesting that logical structures translate more effectively across linguistic boundaries than cultural knowledge.

Limitations and Future Work

The study highlights the high cost of building quality multilingual benchmarks, noting the necessity of human linguistic experts. Furthermore, while the study expanded to 10 languages in supplementary materials, the core benchmark focuses on five. Future work should focus on making the "tone-switching" of NTP intrinsic to model architectures, allowing them to adjust automatically to the cultural domain of a query without external prompting.

Summary updated

BibTeX
@inproceedings{xue2025comprehensive,
  author = {Boyang Xue and Hongru Wang and Weichao Wang and Rui Wang and Sheng Wang and Zeming Liu and Kam-Fai Wong},
  title = {{A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure