When we interact with a traditional software application, we expect determinism: the same input should invariably yield the same output. Large Language Models (LLMs), however, break this fundamental assumption. Because they are probabilistic engines, they do not compute a single "right" answer; instead, they generate text token-by-token by sampling from a shifting probability distribution.
This inherent randomness introduces a pervasive challenge known as stochastic output inconsistency. Even when provided with identical prompts under low temperature settings, an LLM can trace entirely different paths through its token space across separate sessions. While this variability can mimic human creativity in casual conversations, it introduces severe reliability and safety risks when these models are deployed in high-stakes, decision-critical environments.
The Mechanics of Generation Drift
To understand why LLMs are inherently unstable, we must look at how a single token choice alters the trajectory of an entire response. When an LLM processes a prompt, it outputs a probability distribution over its vocabulary for the next token. If the model selects a slightly less probable token due to sampling randomness—or due to subtle hardware-level non-determinism during parallel GPU operations—that token is appended to the prompt.
Because LLMs are autoregressive, this new token changes the context for all subsequent predictions. A tiny divergence at token ten can completely alter the logical trajectory of token one hundred. As noted by the authors of Towards large language model-based personal agents in the enterprise: Current trends and open problems, this variance persists even when models are configured with near-zero temperature settings, presenting a major roadblock for enterprise workflows that require predictable execution.
Where Randomness Meets High-Stakes Reality
The practical consequences of this stochastic behavior span across multiple domains, most notably in healthcare, algorithmic reasoning, and security.
Clinical and Medical Risks
In medicine, diagnostic and dosing consistency is a safety requirement. However, research shows that LLMs frequently exhibit unstable clinical logic. For instance, Aaron Chase et al. demonstrated in Large language models management of complex medication regimens: a case-based evaluation that models can provide entirely different dosing strategies or treatment plans when presented with identical medical scenarios.
This lack of reproducibility is documented across various medical tasks:
- Diagnostics: Tassallah Abdullahi found in Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models that the same clinical prompt can generate different diagnostic results across multiple trials.
- Data Extraction and Categorization: Huibo Yang et al. in Evaluating accuracy and reproducibility of large language model performance on critical care assessments in pharmacy education and Qingxia Wu et al. in Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study highlight that low inter-run reliability prevents these systems from being safely integrated into automated clinical workflows.
- Cross-Lingual Disparity: The instability worsens when queries cross language barriers. Yiqiao Jin et al. developed the XLingEval framework in Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries, revealing that models generate highly inconsistent or even contradictory medical advice when processing the same query across different languages.
Algorithmic and Security Fragility
Stochasticity also undermines tasks requiring mathematical or structural precision. Giorgio Severi et al. evaluated this in Ask and You Shall Receive (a Graph Drawing): Testing ChatGPT's Potential to Apply Graph Layout Algorithms, finding that the probabilistic nature of LLMs introduces errors in algorithmic execution because the same prompt yields different structural results across iterations.
In security frameworks, this variance cuts both ways:
- Defensive Failures: Kaijie Zhu et al. show in MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents that stochastic variance causes parallel execution paths to generate minor variations in tool arguments, leading to false positives or missed detections in comparison-based defenses.
- Adversarial Fragility: Conversely, Junxiao Yang et al. note in Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraints that token-sampling randomness can cause optimized jailbreak prompts to fail unpredictably, as the model may randomly drift away from the intended adversarial distribution.
Quantifying the Instability
Standard evaluation metrics often obscure this problem by measuring "peak" performance—evaluating whether a model can find the right answer on a single run. However, a model that succeeds on the first run but fails on the next nine is highly unstable in production.
To address this gap, Junnan Liu et al. introduced the G-Pass@k metric in Are Your LLMs Capable of Stable Reasoning?. Instead of treating accuracy as a binary pass/fail metric, G-Pass@k quantifies how consistently a model reproduces correct reasoning chains across $k$ independent generations. Their evaluation revealed that many models with high baseline accuracy suffer from severe reasoning instability, forcing users to rely on unpredictable manual regenerations to achieve a correct output.
Harnessing Stochasticity via Consensus Aggregation
If stochasticity is an inescapable property of autoregressive sampling, how can we build reliable systems? A prominent direction in recent research is to stop fighting the randomness and instead use it as a statistical tool. By sampling the model multiple times, we can treat the outputs as a distribution and aggregate them to find a stable consensus.
The Centroid Approach
For structured or decision-making tasks, we can find the statistical "average" of multiple runs. Sana Ebrahimi et al. proposed this strategy with REQUAL-LM in REQUAL-LM: Reliability and Equity through Aggregation in Large Language Models. By treating the black-box LLM as a stochastic sampler, they generate a pool of candidate responses via Monte Carlo sampling and aggregate them into a single "equitable centroid." This process effectively filters out the probabilistic noise of individual runs, yielding an output that reflects the model's stable, core behavior.
Prompt ───► [ LLM Sampler ] ───► Response 1 ┐
───► Response 2 ┼──► [ Centroid Aggregation ] ──► Stable Output
───► Response 3 │
Fine-Grained Synthesis
While voting methods work well for closed-ended tasks (like multiple-choice or math problems), they fail for free-form text generation like summarization or code generation, where two correct answers might not share the same string representation.
To solve this, Xinglin Wang et al. introduced Fine-Grained Self-Consistency (FSC) in Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation. Instead of forcing a vote on the entire text string, FSC breaks down multiple sampled responses into smaller semantic segments. It evaluates the consensus of these individual segments across the generation pool, extracts the most stable and verified components, and synthesizes them into a single, optimized response.
By shifting from single-generation inference to multi-sample consensus, these frameworks transform the inherent weakness of model stochasticity into a mechanism for verification and stability.
Key Foundational Work
Evaluating accuracy and reproducibility of large language model performance on critical care assessments in pharmacy education (2024) — contributed 2 concept elements to this direction.
Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023) — contributed 2 concept elements to this direction.
Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models (2023) — contributed 1 concept elements to this direction.