Beyond accuracy: what AI assurance needs to establish in public services

Louise YorkearXiv 2026 · arXiv

TL;DR

This paper argues that aggregate accuracy is a deceptive metric for AI assurance in public services. By testing GPT-4o on policy classification tasks, the author demonstrates that while accuracy remains stable across configurations, individual labels and substantive conclusions fluctuate significantly, potentially misguiding policymakers. The study calls for governance frameworks that mandate transparency regarding stability, shared errors, and the specific logic used to route AI outputs for human review.

What Problem Does This Paper Address?

Public bodies are increasingly using Large Language Models (LLMs) to classify, summarize, and assess evidence—tasks that shape the information available to decision-makers. The core challenge is predictive multiplicity: different model configurations (or even repeated runs of the same model) can achieve nearly identical aggregate accuracy while assigning different labels to the same individual cases.

If an AI system is used to identify policy priorities or allocate resources, these "hidden" variations matter. A system might be 70% accurate on average, but if different setups disagree on which 30% they get wrong, the resulting "substantive account" (e.g., which policy topics are trending) can shift. Current AI governance, like the EU AI Act and UK guidance, recognizes the need for testing but often leaves the specific metrics and "acceptance thresholds" to the discretion of providers and users without requiring them to justify these choices.

Methods

The author employs a two-pronged approach:

  1. Technical Experiment: Using GPT-4o, the study classifies 1,200 UK Prime Minister’s Questions (PMQs) into 21 policy topics. The author compares an "anchor" configuration against seven alternatives (varying prompts, few-shot examples, and reasoning chains like Chain-of-Thought). To test stability, the anchor was also run five times without changes.
  2. Documentary Analysis: The author analyzed 37 UK and EU policy documents (including the EU AI Act and UK AI Playbook) using a governance framework that distinguishes between decision-making, delivery, coordination, and accountability.

The goal was to see if the technical risks identified in the experiment (like instability and selective evaluation) were addressed by existing or proposed regulations.

Datasets

  • UK Comparative Agendas Project (CAP): A dataset of 9,061 unique PMQs from 1997–2008. The study used a stratified sample of 1,200 questions to ensure representation across imbalanced topic classes (e.g., Government Operations vs. Technology).
  • Human-Coded Labels: The CAP dataset provides human-assigned topic labels, serving as the ground truth reference for calculating accuracy and correctness.

Key Insights and Ideas

  • Agreement $\neq$ Correctness: Configurations often agree with each other more than they agree with human coders. In 191 cases, all eight AI configurations gave the same label, but that label differed from the human reference. Consistency does not guarantee truth.
  • The Accuracy Trap of "Agreement Filtering": If you only keep the questions where all configurations agree, the reported accuracy of the anchor model jumps from 69.6% to 79.0%. This creates an illusion of a better classifier, when in reality, it's just a "selection effect" that ignores difficult cases.
  • Non-Determinism at Temperature Zero: Even with temperature set to zero, identical runs disagreed on 2.78% of labels on average. This baseline "run-to-run" variation is a critical, often overlooked factor in reliability.
  • Fragile Human Review Pipelines: Many frameworks suggest routing "uncertain" outputs to humans. However, the study found that the proportion of outputs flagged for review ranged from 0.9% to 41.0% depending on the configuration, even when using the same score threshold.

Key Results

The main experimental findings are summarized in the table below:

Table 1. Main experimental findings.

Quantitatively, the study found that while accuracy only varied between 68.2% and 70.3% across configurations, a staggering 24.3% of questions received different labels depending on the setup.

![Fig.

  1. Paired label disagreement between each alternative configuration and the anchor.](/figures/art_c8ee3e7f23ce4d34a33972a68203afa5)

As shown in Figure 1, configurations like "Chain-of-thought" and "Reasoning and label" showed the highest disagreement with the anchor (over 10%), far exceeding the baseline run-to-run variation of 2.78%. Qualitatively, these shifts were enough to swap the rankings of the two largest policy topics in the final report, demonstrating how AI configuration choices can directly alter the "broad picture" presented to officials.

Limitations and Future Work

  • Scope: The experiment was limited to one model (GPT-4o) and one specific task (text classification). The author notes that summarization and advice-giving tasks require their own investigations.
  • Model Drift: The repeated runs were conducted five months after the original experiment using unpinned model aliases, meaning some variation might be due to backend model updates rather than inherent stochasticity.
  • Human Element: While the paper discusses routing to human reviewers, it did not test the actual effectiveness of those reviewers in catching AI errors.
  • Future Directions: The author suggests that future work should validate score-extraction methods for human review and examine how these technical variations translate into actual harms or biased public-service decisions.

Summary updated

Abstract

Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.
BibTeX
@misc{yorke2026beyond,
  author = {Louise Yorke},
  title = {{Beyond accuracy: what AI assurance needs to establish in public services}},
  year = {2026},
  eprint = {2611.06655},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06655},
}
Figure