Measuring Determinism in Large Language Models for Software Code Review

Eugene Klishevich, Yegor Denisov-Blanch, Simon Obstbaum, Igor Ciobanu, Michal KosinskiarXiv 2025 · DOI · arXiv

Introduction

Code review is a critical yet resource-intensive phase of software development. As organizations increasingly look toward Large Language Models (LLMs) to automate this process, a fundamental requirement emerges: reliability. For an AI to serve as a trusted reviewer, it must be deterministic. If presented with the same code commit twice, it should ideally provide the same assessment.

While developers typically set the "temperature" parameter to zero to minimize stochasticity, researchers have long suspected that hardware-level floating-point errors and complex model architectures prevent true determinism. This study quantifies this inconsistency within the specific domain of software engineering, investigating the stability of LLM outputs across repeated identical tasks.

Methodology

The researchers implemented a rigorous "test-retest" experimental design using four prominent models: GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, and Llama 3.2 90B Vision. The study utilized 70 Java code commits sourced from a diverse dataset of 1.73 million commits across 108 organizations, ensuring a realistic mix of public and private repository data.

Each model was tasked with answering five specific questions (Q1–Q5) regarding the commits, such as estimating implementation time and judging author experience. To test the impact of context, three prompt lengths were used: Simple (~100 words), Balanced (~350 words), and Explanatory (~750 words). For every combination, the researchers ran the test five times at zero temperature, clearing the context window between runs. Consistency was measured using Pearson’s correlation coefficient across 21,000 individual responses.

The Consistency Gradient

The experiment revealed a clear hierarchy in model reliability. Contrary to expectations of parity among top-tier models, a significant "consistency gradient" was observed. Llama 3.2 90B emerged as the most stable model, while the GPT-4o family showed significantly higher variance.

Heatmap of correlation scores across models, prompt types, and questions.

As shown in the heatmap above, the variance was particularly pronounced in subjective assessments. For instance, on Q5 (Maintainability), Llama 3.2 achieved a perfect 1.00 correlation, whereas GPT-4o dropped to 0.55. This suggests that while some models can maintain internal logic across runs, others "drift" even when the input is identical.

Key Insights

The Prompt Length Paradox

A surprising finding was that more detailed instructions do not necessarily lead to more stable results. For GPT-4o mini, the "Simple" prompt yielded a higher consistency (0.93 for Q1) than the "Explanatory" prompt (0.73). This suggests that excessive detail may provide the model with more "semantic space" to wander, inadvertently increasing output variance.

LLMs vs. Human Baselines

While the lack of perfect determinism is a concern for automation, the study notes that LLMs often remain more consistent than human reviewers. Humans are notoriously subjective, often exhibiting low inter-rater reliability. In this context, even the less consistent LLMs may still provide a more stable baseline than manual human review, despite their internal drift.

Summary of Results

The final rankings based on average correlation scores (where 1.0 represents perfect consistency) highlight the current state of LLM determinism:

Average correlation scores and variance for the four tested models.
  • Llama 3.2 90B: 0.95 (Highest consistency)
  • Claude 3.5 Sonnet: 0.90
  • GPT-4o & GPT-4o mini: 0.72 (Lowest consistency)

Limitations and Future Work

The study focused exclusively on Java commits and a sample size of 70 commits. Furthermore, the research measured precision (consistency) rather than accuracy (validity). A model could theoretically be perfectly consistent while being factually incorrect. Future research will focus on comparing these consistent outputs against ground-truth data from original developers and testing newer reasoning-centric models like OpenAI’s o1.

Summary updated

BibTeX
@misc{klishevich2025measuring,
  author = {Eugene Klishevich and Yegor Denisov-Blanch and Simon Obstbaum and Igor Ciobanu and Michal Kosinski},
  title = {{Measuring Determinism in Large Language Models for Software Code Review}},
  year = {2025},
  eprint = {2502.20747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.20747},
}
Figure