TL;DR
This paper argues that aggregate accuracy is a deceptive metric for AI assurance in public services. By testing GPT-4o on policy classification tasks, the author demonstrates that while accuracy remains stable across configurations, individual labels and substantive conclusions fluctuate significantly, potentially misguiding policymakers. The study calls for governance frameworks that mandate transparency regarding stability, shared errors, and the specific logic used to route AI outputs for human review.
What Problem Does This Paper Address?
Public bodies are increasingly using Large Language Models (LLMs) to classify, summarize, and assess evidence—tasks that shape the information available to decision-makers. The core challenge is predictive multiplicity: different model configurations (or even repeated runs of the same model) can achieve nearly identical aggregate accuracy while assigning different labels to the same individual cases.
If an AI system is used to identify policy priorities or allocate resources, these "hidden" variations matter. A system might be 70% accurate on average, but if different setups disagree on which 30% they get wrong, the resulting "substantive account" (e.g., which policy topics are trending) can shift. Current AI governance, like the EU AI Act and UK guidance, recognizes the need for testing but often leaves the specific metrics and "acceptance thresholds" to the discretion of providers and users without requiring them to justify these choices.
Methods
The author employs a two-pronged approach:
- Technical Experiment: Using GPT-4o, the study classifies 1,200 UK Prime Minister’s Questions (PMQs) into 21 policy topics. The author compares an "anchor" configuration against seven alternatives (varying prompts, few-shot examples, and reasoning chains like Chain-of-Thought). To test stability, the anchor was also run five times without changes.
- Documentary Analysis: The author analyzed 37 UK and EU policy documents (including the EU AI Act and UK AI Playbook) using a governance framework that distinguishes between decision-making, delivery, coordination, and accountability.
The goal was to see if the technical risks identified in the experiment (like instability and selective evaluation) were addressed by existing or proposed regulations.
Datasets
- UK Comparative Agendas Project (CAP): A dataset of 9,061 unique PMQs from 1997–2008. The study used a stratified sample of 1,200 questions to ensure representation across imbalanced topic classes (e.g., Government Operations vs. Technology).
- Human-Coded Labels: The CAP dataset provides human-assigned topic labels, serving as the ground truth reference for calculating accuracy and correctness.
Key Insights and Ideas
- Agreement $\neq$ Correctness: Configurations often agree with each other more than they agree with human coders. In 191 cases, all eight AI configurations gave the same label, but that label differed from the human reference. Consistency does not guarantee truth.
- The Accuracy Trap of "Agreement Filtering": If you only keep the questions where all configurations agree, the reported accuracy of the anchor model jumps from 69.6% to 79.0%. This creates an illusion of a better classifier, when in reality, it's just a "selection effect" that ignores difficult cases.
- Non-Determinism at Temperature Zero: Even with temperature set to zero, identical runs disagreed on 2.78% of labels on average. This baseline "run-to-run" variation is a critical, often overlooked factor in reliability.
- Fragile Human Review Pipelines: Many frameworks suggest routing "uncertain" outputs to humans. However, the study found that the proportion of outputs flagged for review ranged from 0.9% to 41.0% depending on the configuration, even when using the same score threshold.
Key Results
The main experimental findings are summarized in the table below:
Quantitatively, the study found that while accuracy only varied between 68.2% and 70.3% across configurations, a staggering 24.3% of questions received different labels depending on the setup.

As shown in Figure 1, configurations like "Chain-of-thought" and "Reasoning and label" showed the highest disagreement with the anchor (over 10%), far exceeding the baseline run-to-run variation of 2.78%. Qualitatively, these shifts were enough to swap the rankings of the two largest policy topics in the final report, demonstrating how AI configuration choices can directly alter the "broad picture" presented to officials.
Limitations and Future Work
- Scope: The experiment was limited to one model (GPT-4o) and one specific task (text classification). The author notes that summarization and advice-giving tasks require their own investigations.
- Model Drift: The repeated runs were conducted five months after the original experiment using unpinned model aliases, meaning some variation might be due to backend model updates rather than inherent stochasticity.
- Human Element: While the paper discusses routing to human reviewers, it did not test the actual effectiveness of those reviewers in catching AI errors.
- Future Directions: The author suggests that future work should validate score-extraction methods for human review and examine how these technical variations translate into actual harms or biased public-service decisions.
Summary updated