Introduction
Code review is a critical yet resource-intensive phase of software development. As organizations increasingly look toward Large Language Models (LLMs) to automate this process, a fundamental requirement emerges: reliability. For an AI to serve as a trusted reviewer, it must be deterministic. If presented with the same code commit twice, it should ideally provide the same assessment.
While developers typically set the "temperature" parameter to zero to minimize stochasticity, researchers have long suspected that hardware-level floating-point errors and complex model architectures prevent true determinism. This study quantifies this inconsistency within the specific domain of software engineering, investigating the stability of LLM outputs across repeated identical tasks.
Methodology
The researchers implemented a rigorous "test-retest" experimental design using four prominent models: GPT-4o, GPT-4o mini, Claude 3.5 Sonnet, and Llama 3.2 90B Vision. The study utilized 70 Java code commits sourced from a diverse dataset of 1.73 million commits across 108 organizations, ensuring a realistic mix of public and private repository data.
Each model was tasked with answering five specific questions (Q1–Q5) regarding the commits, such as estimating implementation time and judging author experience. To test the impact of context, three prompt lengths were used: Simple (~100 words), Balanced (~350 words), and Explanatory (~750 words). For every combination, the researchers ran the test five times at zero temperature, clearing the context window between runs. Consistency was measured using Pearson’s correlation coefficient across 21,000 individual responses.
The Consistency Gradient
The experiment revealed a clear hierarchy in model reliability. Contrary to expectations of parity among top-tier models, a significant "consistency gradient" was observed. Llama 3.2 90B emerged as the most stable model, while the GPT-4o family showed significantly higher variance.
As shown in the heatmap above, the variance was particularly pronounced in subjective assessments. For instance, on Q5 (Maintainability), Llama 3.2 achieved a perfect 1.00 correlation, whereas GPT-4o dropped to 0.55. This suggests that while some models can maintain internal logic across runs, others "drift" even when the input is identical.
Key Insights
The Prompt Length Paradox
A surprising finding was that more detailed instructions do not necessarily lead to more stable results. For GPT-4o mini, the "Simple" prompt yielded a higher consistency (0.93 for Q1) than the "Explanatory" prompt (0.73). This suggests that excessive detail may provide the model with more "semantic space" to wander, inadvertently increasing output variance.
LLMs vs. Human Baselines
While the lack of perfect determinism is a concern for automation, the study notes that LLMs often remain more consistent than human reviewers. Humans are notoriously subjective, often exhibiting low inter-rater reliability. In this context, even the less consistent LLMs may still provide a more stable baseline than manual human review, despite their internal drift.
Summary of Results
The final rankings based on average correlation scores (where 1.0 represents perfect consistency) highlight the current state of LLM determinism:
- Llama 3.2 90B: 0.95 (Highest consistency)
- Claude 3.5 Sonnet: 0.90
- GPT-4o & GPT-4o mini: 0.72 (Lowest consistency)
Limitations and Future Work
The study focused exclusively on Java commits and a sample size of 70 commits. Furthermore, the research measured precision (consistency) rather than accuracy (validity). A model could theoretically be perfectly consistent while being factually incorrect. Future research will focus on comparing these consistent outputs against ground-truth data from original developers and testing newer reasoning-centric models like OpenAI’s o1.
Summary updated