Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text

George KoomullilarXiv 2026 · arXiv

This paper tackles a high-stakes problem in administrative data: how do we use AI to count money when the "ground truth" is a moving target? Specifically, the author explores AI-assisted measurement of research funding in US tax filings, where a single text description (e.g., "Cancer research") might apply to thousands of individual grant records.

TL;DR

The paper provides a rigorous framework for using LLMs to estimate monetary totals from administrative text, even when human experts disagree on the labels. By combining prediction-powered inference with active human review, the author demonstrates how to reduce measurement error by 28% while maintaining statistical validity under complex sampling designs.

What Problem Does This Paper Address?

In many administrative datasets, the unit we read (a text description) is not the unit we count (a dollar amount). For example, a university might receive 10,000 grants, but many share the same purpose text. If an AI mislabels one frequent text, the error propagates to millions of dollars.

Standard machine learning metrics like "accuracy" are insufficient here because an error on a \$10M grant is far more damaging than an error on a \$100 grant. Furthermore, human "gold standards" are often anything but—different human readers frequently disagree on how to categorize ambiguous text. This paper addresses three critical decisions for an organization using AI for such tasks:

  1. Which machine labels can be accepted automatically without human review?
  2. Which texts should a human expert prioritize reading given a limited budget?
  3. What is the final total, and how much should we trust it given that even humans disagree?

Methods

The author builds a "testbed" using 121,070 distinct grant-purpose texts representing \$126.4B in funding. The core methodology involves:

  • Calibrated Aggregation: Using a ridge multinomial logistic calibrator to ensemble a rule-based classifier and three LLMs (Claude 5.5, Gemini 2.5 Flash, and GPT-5 mini).
  • Prediction-Powered Inference (PPI): Specializing design-based PPI to "rectify" machine predictions using a small human-labeled sample. The paper proves conditions for exact unbiasedness under Poisson sampling.
  • Selective Labelling: Implementing Learn-then-Test (LTT) and weighted conformal p-values to identify a subset of machine labels that meet a specific dollar-weighted error target (e.g., "I am 95% sure the error in these accepted labels is less than 5% of the total dollars").
  • Active Statistical Inference: Using Neyman allocation to prioritize human review for texts that have high dollar volume and high predicted uncertainty.

The data flow and label usage are summarized in Figure 1:

Figure 1: Data flow and label use. The human labels enter the calibrator (cross-fitted), each task’s rectifier or rule, and each task’s check; no text’s own label enters its own probability.

Datasets

  • Primary Dataset: 799,675 grant records (\$126.4B) from US IRS Form 990 filings (tax years 2020–2024).
  • Human Sample: A designed sample of 651 texts read independently by three different people. This allows the author to re-run the entire pipeline under different "truths" to see how sensitive the results are to human disagreement.

Key Insights and Ideas

  • The Stratification Paradox: The author finds that if you already stratify your human sample based on machine "votes" (e.g., "all models agree this is research"), the marginal gain from using a complex PPI "rectifier" model is small. The stratification itself captures most of the machine's value.
  • Dollar Measure vs. Text Measure: Error control for counts (number of texts) is powerless when applied to dollars. A rule that works for counts might accept zero texts when asked to guarantee dollar-weighted error because the weights of rare, high-value texts become too volatile.
  • Human Disagreement is the Ceiling: The spread between different human readers (\$3.93B) was 2.2 times larger than the sampling interval for a single reader (\$1.77B). This suggests that the "bias" of a specific human codebook interpreter is a larger source of uncertainty than the statistical sampling itself.

Key Results

  • Error Reduction: A two-phase active review policy reduced the Root Mean Square Error (RMSE) of the total by 28% at a budget of 100 human-labeled texts compared to standard dollar-proportional sampling.
  • Bias Rectification: Eight different predictors with raw totals ranging from \$11.2B to \$15.0B were brought into agreement within a narrow \$0.4B range once rectified by the human sample.
  • Selective Labelling Yield: The Learn-then-Test rule successfully accepted 80% of unsampled dollars at a 5% error target ($\alpha = 0.05$), significantly reducing the manual workload.

Limitations and Future Work

  • Reference Dependency: All validity claims are relative to a specific human reader. The paper shows that "truth" is subjective, and future work is needed on adjudicating splits between disagreeing experts.
  • Small-Area Under-coverage: While national totals were robust, estimates for individual institutions (e.g., a specific university's research total) often "under-covered," meaning the confidence intervals were too narrow.
  • Finite-Sample Bounds: The Learn-then-Test application currently relies on asymptotic approximations for unequal weights; the author notes that a truly "certified" rule would require new finite-sample bounds tailored to this specific sampling design.

Summary updated

Abstract

We study AI-assisted measurement when one text label applies to many administrative records, the target is a monetary total, and the human readings that anchor the machines disagree. The testbed is 121,070 grant-purpose texts from US tax filings (799,675 records, \$126.4B), labelled by a rule-based classifier and three language models, with a designed sample of 651 texts read independently by three people under one codebook. We state exact conditions for a fixed-predictor difference estimator and for a cross-fitted predictor whose folds are assigned to the whole frame before sampling, under Poisson sampling, and assess the implemented fixed-size procedures by simulation: 8 predictors whose own totals span \$11.2-15.0B agree to within \$0.4B once rectified. Two rules that accept machine labels at a stated dollar-weighted error, weighted conformal p-values under a dollar measure and Learn-then-Test on the design estimate of the error, are evaluated on nested held-out folds and on re-draws of the design with synthetic truths; their error control in deployment is not established, and the count-based rule accepts no text under a dollar-drawn sample. In a replay that charges every human label, a two-phase review policy whose calibrator is fitted on its own first phase reduces the RMSE of the total by 28% at a budget of 100 texts relative to dollar-proportional sampling, while pure Neyman allocation fails with an uncalibrated predictor. Independent readers shift the estimated total by \$3.93B, 2.2 times the within-reader sampling interval, and institution-level intervals under-cover at every size. Every headline result is re-run under each reader. The results identify both the benefits and the limits of using learned predictions to decide what a person should verify.
BibTeX
@misc{koomullil2026counting,
  author = {George Koomullil},
  title = {{Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text}},
  year = {2026},
  eprint = {2611.13215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13215},
}
Figure