This paper tackles a high-stakes problem in administrative data: how do we use AI to count money when the "ground truth" is a moving target? Specifically, the author explores AI-assisted measurement of research funding in US tax filings, where a single text description (e.g., "Cancer research") might apply to thousands of individual grant records.
TL;DR
The paper provides a rigorous framework for using LLMs to estimate monetary totals from administrative text, even when human experts disagree on the labels. By combining prediction-powered inference with active human review, the author demonstrates how to reduce measurement error by 28% while maintaining statistical validity under complex sampling designs.
What Problem Does This Paper Address?
In many administrative datasets, the unit we read (a text description) is not the unit we count (a dollar amount). For example, a university might receive 10,000 grants, but many share the same purpose text. If an AI mislabels one frequent text, the error propagates to millions of dollars.
Standard machine learning metrics like "accuracy" are insufficient here because an error on a \$10M grant is far more damaging than an error on a \$100 grant. Furthermore, human "gold standards" are often anything but—different human readers frequently disagree on how to categorize ambiguous text. This paper addresses three critical decisions for an organization using AI for such tasks:
- Which machine labels can be accepted automatically without human review?
- Which texts should a human expert prioritize reading given a limited budget?
- What is the final total, and how much should we trust it given that even humans disagree?
Methods
The author builds a "testbed" using 121,070 distinct grant-purpose texts representing \$126.4B in funding. The core methodology involves:
- Calibrated Aggregation: Using a ridge multinomial logistic calibrator to ensemble a rule-based classifier and three LLMs (Claude 5.5, Gemini 2.5 Flash, and GPT-5 mini).
- Prediction-Powered Inference (PPI): Specializing design-based PPI to "rectify" machine predictions using a small human-labeled sample. The paper proves conditions for exact unbiasedness under Poisson sampling.
- Selective Labelling: Implementing Learn-then-Test (LTT) and weighted conformal p-values to identify a subset of machine labels that meet a specific dollar-weighted error target (e.g., "I am 95% sure the error in these accepted labels is less than 5% of the total dollars").
- Active Statistical Inference: Using Neyman allocation to prioritize human review for texts that have high dollar volume and high predicted uncertainty.
The data flow and label usage are summarized in Figure 1:
Datasets
- Primary Dataset: 799,675 grant records (\$126.4B) from US IRS Form 990 filings (tax years 2020–2024).
- Human Sample: A designed sample of 651 texts read independently by three different people. This allows the author to re-run the entire pipeline under different "truths" to see how sensitive the results are to human disagreement.
Key Insights and Ideas
- The Stratification Paradox: The author finds that if you already stratify your human sample based on machine "votes" (e.g., "all models agree this is research"), the marginal gain from using a complex PPI "rectifier" model is small. The stratification itself captures most of the machine's value.
- Dollar Measure vs. Text Measure: Error control for counts (number of texts) is powerless when applied to dollars. A rule that works for counts might accept zero texts when asked to guarantee dollar-weighted error because the weights of rare, high-value texts become too volatile.
- Human Disagreement is the Ceiling: The spread between different human readers (\$3.93B) was 2.2 times larger than the sampling interval for a single reader (\$1.77B). This suggests that the "bias" of a specific human codebook interpreter is a larger source of uncertainty than the statistical sampling itself.
Key Results
- Error Reduction: A two-phase active review policy reduced the Root Mean Square Error (RMSE) of the total by 28% at a budget of 100 human-labeled texts compared to standard dollar-proportional sampling.
- Bias Rectification: Eight different predictors with raw totals ranging from \$11.2B to \$15.0B were brought into agreement within a narrow \$0.4B range once rectified by the human sample.
- Selective Labelling Yield: The Learn-then-Test rule successfully accepted 80% of unsampled dollars at a 5% error target ($\alpha = 0.05$), significantly reducing the manual workload.
Limitations and Future Work
- Reference Dependency: All validity claims are relative to a specific human reader. The paper shows that "truth" is subjective, and future work is needed on adjudicating splits between disagreeing experts.
- Small-Area Under-coverage: While national totals were robust, estimates for individual institutions (e.g., a specific university's research total) often "under-covered," meaning the confidence intervals were too narrow.
- Finite-Sample Bounds: The Learn-then-Test application currently relies on asymptotic approximations for unequal weights; the author notes that a truly "certified" rule would require new finite-sample bounds tailored to this specific sampling design.
Summary updated