Audit Where Raters Disagree: Sector-Specific Debiased Inference for LLM-Read ESG Disclosures

Regulators, investors and researchers increasingly let large language models read corporate sustainability reports and record what each firm discloses. They then treat those machine readings as data, computing sector-level disclosure rates or running regressions on them. The machine reading is a proxy, and its errors are not random. Firms that know more about their environmental footprint than outsiders do have reasons to write vague, confident prose that sounds like disclosure without committing to numbers. An extractor tends to credit that prose as "disclosed." This kind of error is also concentrated by sector, because what counts as a material disclosure differs between a bank and a chemical plant. This note proposes to estimate sector-specific disclosure statistics from cheap machine readings plus a small, randomized human audit. The audit is concentrated where the information gap between firm and outsider is largest. We measure that gap with a signal that already exists before any audit: disagreement among commercial ESG raters.

In two sentences, the idea is this. Debiasing methods such as prediction-powered inference (PPI, which corrects a machine-label average with a small human-labeled sample) already work when the strata are well chosen. The open question is how to choose strata when proxy errors are strategic and sector-dependent. We hypothesize that cross-rater divergence crossed with sector predicts where LLM extraction fails better than the extractor's own confidence does. If that is true, allocating human audits by divergence buys valid, narrower sector-level intervals at a fixed budget.

Core Claim

Two ingredients are new here, and both can fail. The first is a measurement hypothesis. Errors in LLM disclosure extraction concentrate in low-transparency firms and in sectors whose material metrics are hard to quantify. Those errors are often confident, so stratifying on extractor confidence, the default in Stratified Prediction-Powered Inference, will miss them. Rater divergence will not miss them. The second is a design. Audits are allocated Neyman-style (more audits to strata whose residual error varies more) over sector by divergence strata. Each SASB sub-industry gets its own rectification, and we check per sector that no sector's interval is worse than under pooled correction.

Validity does not depend on the hypothesis. Audit inclusion probabilities are set by us and known, so the design-based correction stays unbiased whether or not divergence is informative. Only efficiency is at stake. That makes the hypothesis cheap to test and safe to deploy.

Why This Intersection, and What Already Exists

Four bodies of work meet here, and each plays a different role.

Environmental information asymmetry supplies the stratifier. Empowerment or detriment studies Chinese A-share firms from 2009 to 2022 and finds that disagreement among six ESG raters carries information rather than noise. Its effect on green institutional investors is concentrated in low-transparency firms (coefficient 0.203) and low-ESG firms (0.357). That paper uses econometric regressions only and never asks why raters diverge at the text level. We use its finding as motivation: divergence marks firms where outsiders cannot tell what is true. Those are the firms where a text-reading proxy should be least trustworthy.

Sector-specific regulatory design supplies the estimand. The argument in Regulação da Inteligência Artificial is that monolithic rules fail across heterogeneous sectors. Similarly, evidence that environmental metrics matter little for financial firms but are central for heavy industry (Liu et al.) implies that a regulator needs per-sector numbers against per-sector metric lists. A single pooled disclosure rate is not enough. EulerESG already builds an extractor aligned to SASB (Sustainability Accounting Standards Board) sub-industry standards and reports 0.95 average extraction accuracy for GPT-5. It treats those outputs as ground truth and attaches no uncertainty to them. We reuse its extraction design as the proxy. What we add is the inferential layer it lacks.

Proxy label bias supplies both the warning and the machinery. Using Imperfect Surrogates for Downstream Inference shows that plugging LLM labels into downstream analyses can drop nominal 95% coverage to 40%. It also shows that design-based correction with known sampling probabilities restores coverage, but its method is formulated only for proxies used as outcomes. PPI++ adds power tuning so that corrected intervals are never wider than audit-only ones, using a single pooled tuning parameter. M-estimation under Two-Phase Multiwave Sampling handles general estimating equations under adaptive audit waves. It assumes proxy errors are passive noise and notes undercoverage for quantiles on small audit samples. StratPPI is the closest prior work. It stratifies with per-stratum rectification and Neyman allocation, but its strata are fixed in advance, it was tested only on NLP evaluation, and it becomes unstable when strata are many relative to budget. Our contribution is not a new estimator. It is a tested answer to the question StratPPI leaves to the user, namely which strata to use, in a domain where errors are strategic. We also extend the setting to sector-specific estimands, including a case where the proxy enters a regression as an explanatory variable.

Subgroup-specific modeling supplies the evaluation standard. Fairness without Harm asks that every group prefer its own decoupled model to the pooled one, a property it calls rationality. We borrow that notion as a per-sector no-harm check on interval width. Understanding challenges to the interpretation of disaggregated evaluations shows that subgroup gaps can reflect a true difference in the outcome given the features, which it calls outcome shift. It shows this on ACSPublicCoverage and finds that reweighting does not close the gap. That is our reason to rectify within each sector rather than reweight a pooled correction. The relationship between disclosed text and the true disclosure state plausibly differs by sector.

Research Program

We first build the proxy. An EulerESG-style extractor, using SASB sub-industry metric lists with keyword and semantic retrieval, labels each firm-metric pair as disclosed with numbers, discussed qualitatively, or absent. We harden it with the four-step prompting from ESG-Bench, which raised LLaMA-3.2 from 76% to 96% accuracy on that benchmark. We use the strongest extractor available so that any remaining proxy bias cannot be blamed on a weak model.

We then build the gold standard. Trained annotators read the cited report passages and decide whether a quantitative, metric-specific disclosure actually exists. This construct is what the "disclosed" proxy is meant to measure. The firm population is the CSMAR A-share panel used in Empowerment or detriment, because its six-rater divergence measure is already defined there. Its reports are mostly Chinese, which also tests the extractor beyond EulerESG's English-only scope. We collect a deliberately oversized audit of several thousand firm-metric pairs across sub-industries. Most of that audit serves as a known-truth pool for resampling experiments. A small held-out subset of reports overlapping ESG-Bench lets us check hallucination behavior against its human annotations.

The estimator is stratified prediction-powered estimation with per-stratum power tuning. For stratum $k$ with $N_k$ machine-read pairs, an audited subset $A_k$ of size $n_k$, proxy reading $f(x)$ and audited value $y$, it is

$$\hat\theta_k = \lambda_k \frac{1}{N_k}\sum_{i=1}^{N_k} f(x_i) + \frac{1}{n_k}\sum_{j\in A_k}\big(y_j - \lambda_k f(x_j)\big),$$

where $\lambda_k\in[0,1]$ is chosen within stratum to minimize variance, as in PPI++. Sector estimates combine strata by their population weights. Audits are allocated so that $n_k \propto N_k \sigma_k$, where $\sigma_k$ is the standard deviation of the residual $y-\lambda_k f(x)$, estimated from a small pilot wave. This is the standard Neyman rule for minimizing variance at a fixed total budget.

Strata are sub-industry crossed with divergence tercile. Where a sub-industry is too small, we merge across divergence terciles before merging across sectors. This addresses the small-stratum instability StratPPI flags.

For the downstream regression, we re-estimate the Empowerment paper's question with LLM-measured disclosure completeness as an explanatory variable. We treat true completeness as a phase-two variable inside the two-phase M-estimation framework of the multiwave paper. This is the proxy-as-covariate case that DSL does not cover.

Baselines

The naive reference is surrogate-only estimation: take EulerESG-style extractions as truth and compute sector disclosure rates and regression coefficients from them. This is current practice in the systems we cite. It shows how large the problem is, and on social-science data it covered the truth only 40% of the time (Using Imperfect Surrogates).

The second reference is audit-only classical estimation from the same randomized human sample. Every corrected method must beat it on interval width at nominal coverage.

The pooled competitor is PPI++ with a single tuning parameter. The file gives no reported number for its deforestation results ("narrower intervals than classical and standard PPI").

The strongest existing competitor is StratPPI. We run it in two forms because each is a plausible user default. One stratifies by sub-industry alone. The other stratifies by extractor confidence, which is the option the StratPPI paper itself proposes. The best number reported for StratPPI is a further 10% reduction in confidence interval width over unstratified PPI on Galaxy Zoo, with 300 human labels giving the precision of 600. We found no reported result on Lacuna for any debiased estimator on ESG disclosure data.

We also run the approximately greedy multiwave estimator. It reports effective sample size gains up to 1.8x with nominal 90% coverage on forest cover and AlphaFold.

The key ablation holds everything fixed and swaps only the stratifying variable among four options: sector, confidence, divergence, and sector crossed with divergence.

Every method gets the same machine readings, the same audit budget (we sweep 200, 500 and 1,000 audited pairs), and the same pilot wave for any allocation tuning. Results average over repeated resamples from the known-truth pool. Beating StratPPI means narrower average sector-level intervals at equal or better coverage. The threshold is set at the size of StratPPI's own reported gain over PPI, about 10% narrower. The per-sector check must also pass: no sub-industry's interval is wider than under pooled PPI++.

Why not just…?

Why not just stratify on the extractor's confidence, as StratPPI already allows? This is the main threat, and if it matches divergence stratification, the proposal reduces to applying StratPPI in a new domain. The argument against it is mechanistic. Strategic vagueness produces fluent text that the extractor reads confidently, so confidence tracks ambiguity in the text rather than the gap between text and substance. The stratifier ablation settles this directly. We will also report how much of the residual error variance each candidate stratifier explains in the audit pool. If confidence explains as much, we will say so and recommend it.

Why not just make the extractor more accurate? ESG-Bench's prompting already reaches 96%. DSL's results show that accuracy does not guarantee valid inference when errors correlate with the variables of interest. The remaining 4% matters most if it falls disproportionately on low-transparency firms, which is exactly where the information effect in Empowerment or detriment lives. Our surrogate-only baseline uses the best extractor, so this alternative is measured rather than argued away.

Why not just let adaptive multiwave sampling find the bad regions on its own? This is a real competitor, since its whole purpose is to send audits where proxy error is volatile. Its weakness here is budget. With dozens of sub-industries and a few hundred audits, an early wave sees too few errors per region to locate them, and the authors already report density-estimation trouble on small samples. Divergence is free and available before wave one, so it can serve as that estimator's starting allocation. We test multiwave alone, divergence strata alone, and the combination. If multiwave alone wins at every budget, divergence has no role to play.

Why not just pool all sectors and add sector indicators? A sector intercept shifts the mean correction. It cannot let the weight placed on the proxy differ by sector, and that weight is what the outcome-shift evidence in the disaggregated-evaluation paper suggests should vary. The per-sector no-harm check against pooled PPI++ shows whether sector-specific tuning is needed.

Decisive Checks

Three outcomes decide the project. The first is coverage. Using the known-truth pool, every corrected method must hold nominal 90% and 95% coverage within each sub-industry, not just on average. A failure in small sectors would show that our stratum-merging rule is inadequate.

The second is the measurement hypothesis itself. Audited error rates should rise with rater divergence within a sector more steeply than they rise with falling extractor confidence. If they do not, the stratifier claim is false.

The third is substantive. We re-estimate the rater-divergence regression with debiased disclosure completeness as a covariate. We then report whether the concentration of the information effect in low-transparency firms survives, shrinks, or grows once measurement error in the very firms it concerns is corrected. Either answer matters to regulators deciding which sectors need third-party verification rather than self-reporting.