Introduction
Large Language Models (LLMs) are frequently overconfident, often presenting hallucinations with absolute certainty. This research introduces the concept of anthropomimetic uncertainty, suggesting that for AI to be truly trustworthy, its communication of doubt must evolve beyond simple confidence scores. Instead, models should emulate the nuanced, context-aware linguistic behaviors humans employ to signal uncertainty.
While the goal is for LLMs to be honest about their knowledge gaps, they remain "confidently wrong" in many scenarios. Current research in Natural Language Processing (NLP) has focused primarily on "verbalized uncertainty"—requesting models to output a percentage or numerical score (e.g., "I am 80% sure"). This paper argues that such a narrow focus ignores the social and contextual nature of human uncertainty.
Methodology
The researchers employed a multi-pronged approach to diagnose the current state of LLM uncertainty:
- Literature Survey: An analysis of 53 recent publications to categorize how the field elicits uncertainty.
- Corpus Analysis: A frequency analysis of massive datasets (News, Reddit, ArXiv) to observe natural human use of numbers and "epistemic markers" (e.g., certainly vs. possibly).
- Social Role Simulation: Testing six commercial LLMs across 250 TriviaQA questions while assigning different "social roles" (e.g., Employee to Boss) to observe shifts in confidence.
- Calibration Metrics: Using Expected Calibration Error (ECE) for numerical confidence and Yule’s Y to measure "linguistic calibration"—the correlation between a model’s verbal expressions of certainty and its actual correctness.
Analysis of Current Research Trends
The literature survey highlights a significant imbalance in how uncertainty is studied. As illustrated in the following figures, the field is dominated by simple prompting and numerical outputs.
Specifically, 61% of current research relies on numerical registers (scales like 0-1 or 1-100), while only 19% explores "fluent verbalization," where uncertainty is integrated naturally into the model's response.
Key Insights
- The "Round Number" Bias: LLMs mimic training data rather than performing pure calculations. Humans often use numbers like "50%" or "100%" as linguistic placeholders for "maybe" or "definitely." LLMs inherit this bias, leading to skewed probability distributions.
- Social Hierarchy Distorts Truth: When assigned hierarchical roles (e.g., a Citizen speaking to a Monarch), models are significantly more likely to express high certainty, even if their underlying accuracy remains unchanged. The social role effectively overrides the model's internal reasoning.
- The Sycophancy Trap: Models trained via RLHF are biased toward "helpfulness." Since human evaluators often prefer confident answers, models learn to downplay uncertainty to satisfy the user.
- Domain-Specific Calibration: Uncertainty is not a global property; a model may be well-calibrated in one domain (e.g., Astronomy) but wildly overconfident in another (e.g., World Religions).
Conclusion and Future Directions
Current LLM uncertainty lacks social and linguistic nuance. Future work should move toward explaining uncertainty—helping users understand why a model is unsure—and addressing the English-centric bias in current datasets. Furthermore, personalizing uncertainty communication based on a user's specific risk tolerance will be essential for developing truly anthropomimetic and trustworthy AI.
Summary updated