Measuring How Jurisdictions Shape Proxy Labels

Many high-stakes prediction systems train on an administrative record that stands in for what we actually care about. They use an arrest as a stand-in for an offense, or a medical bill as a stand-in for medical need. A record like this is produced by an institution: a police agency that clears some cases and not others, a health system that spends more on some patients than on others. Institutions differ across cities, states and countries, so the same proxy label can mean different things in different places. A model trained on pooled data, or exported from one jurisdiction to another, inherits those differences without anyone noticing. This note proposes to measure, rather than assume, how much each jurisdiction changes the probability that a fixed underlying event becomes a recorded label, and how much that change differs between demographic groups.

The tool is borrowed from observational causal inference. We treat "which jurisdiction handled this case" as the treatment. We reweight cases so that the case mix, meaning crime type, victim characteristics, offender age and similar features, is balanced across jurisdictions. Any remaining gap in arrest rates then estimates what the jurisdiction itself does to the label. The estimates have two uses. They give comparative policy analysis an outcome-based measure of what a regime does to the data its algorithms learn from, which is different from what its statutes say. They also tell a modeler whether training separate models per demographic group will help, or will instead specialize each model to its group's recording bias.

Core Claim

When some record of the underlying event exists independently of the proxy, the proxy-generation process can be separated from case mix by covariate balancing across jurisdictions. In crime data, the event record is an incident reported to police, and the proxy is whether that incident led to an arrest. We expect two things from this separation. First, much of the cross-jurisdiction "outcome shift" that current diagnostics attribute to different feature–outcome relationships is in fact a shift in how labels get recorded. Second, in exactly those settings, subgroup-specific models trained on the proxy will look better on the proxy and worse on the construct.

Let $i$ index recorded incidents, $X_i$ the case-mix features, $G_i$ the offender's demographic group, $J_i$ the handling jurisdiction, and $A_i$ the proxy label (arrest). Write $\mu_{k,g}(x) = P(A=1 \mid X=x, G=g, J=k)$. The proxy-generation effect of jurisdiction $j$ relative to a reference $r$, for group $g$, is

$$\tau_{j,g} = \mathbb{E}_{X \sim P(X \mid J=j,\, G=g)}\left[\mu_{j,g}(X) - \mu_{r,g}(X)\right],$$

which asks how often group-$g$ cases of the kind jurisdiction $j$ actually sees would have produced a label there versus in the reference. The differential recording effect between two groups is $\delta_j = \tau_{j,g_1} - \tau_{j,g_2}$. We estimate $\tau_{j,g}$ with a doubly robust estimator that combines jurisdiction propensity weights, $e_k(x,g) = P(J=k \mid X=x, G=g)$, with cross-fitted outcome models. "Doubly robust" means the estimate is consistent if either the propensity model or the outcome model is correct.

The identifying assumption deserves a name. Call it case ignorability: conditional on $X$ and $G$, incidents in different jurisdictions do not differ in the latent strength of the underlying event. This assumption is untestable, so the program pairs every estimate with a sensitivity analysis.

Prior Work and Where This Sits

Two empirical papers motivate the problem. Dissecting racial bias in an algorithm used to manage the health of populations showed that using cost as a proxy for need left Black patients sicker at the same risk score. Relabeling raised Black enrollment from 17.7% to 46.5%. That study concerns one health system, so the proxy mapping is fixed. On the Validity of Arrest as a Proxy for Offense used 3.4 million NIBRS (National Incident-Based Reporting System) incidents. It showed that aggregate racial gaps in arrest come from agency-level clearance differences. Memphis had an 89% Black offender share and a 26% arrest rate; Nashville had 57% and 54%. Within-city racial gaps were about 4%. That paper is diagnostic and regression-based. It does not define a jurisdiction-level estimand, does not enforce covariate overlap, and does not link its findings to policy or to model design. We reuse its data and its geographic insight, and we turn the diagnosis into an estimation procedure.

The closest methodological neighbor is Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness. It reweights a population to match a subgroup's features in order to tell covariate shift from outcome shift. On ACSPublicCoverage it finds gaps that reweighting cannot close, and it recommends subgroup-aware models. It treats the recorded label as valid. We extend its reweighting across jurisdictions and split its "outcome shift" into a construct part and a label-recording part. That split decides whether its recommendation holds.

Three papers constrain the design. Fair prediction with disparate impact shows that with unequal base rates, calibration and error-rate balance cannot both hold, so parity constraints against a tainted label cannot fix it. Using Imperfect Surrogates for Downstream Inference shows that treating surrogate labels as truth can collapse 95% confidence intervals to 40% coverage. Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift builds doubly robust proxy-plus-gold estimators. Both of the last two need gold labels and assume the feature–label relationship is the same in labeled and unlabeled data, which is exactly what fails when recording varies by jurisdiction.

On the modeling side, Fairness without Harm: Decoupled Classifiers with Preference Guarantees trains per-group classifiers that each group prefers to the pooled one, with up to a 30.8% accuracy gain on apnea. Diversified Ensembling aggregates subgroup specialists. Both judge success on the observed label, which is the failure mode we test.

Robust Propensity Score Computation Method based on Machine Learning with Label-corrupted Data handles corrupted treatment labels. It leaves corrupted outcome labels unaddressed, and outcome labels are our focus.

On the policy side, Navigating Governance Paradigms and Explainability in AI Policies compare regimes by what their texts cover. The first finds that EU rules cover 92% of mapped processes and the UK under 35%. Neither measures what regimes do to data. Outsider Oversight calls for concrete audit targets. Our estimand is offered as one such target: a measure an auditor could compute.

Research Program

The first component is the estimator and its diagnostics. We will build multi-valued jurisdiction propensity models within each group, using gradient-boosted trees with cross-fitting, and report overlap explicitly. Jurisdiction pairs with poor overlap get no estimate rather than an extrapolated one. Because a positivity failure would otherwise silently change the target, we report how much of the population trimming drops.

For sensitivity, we adapt a standard bounded-confounding analysis. It asks how strongly an unmeasured case-severity variable would have to differ between jurisdictions to explain $\tau_{j,g}$ away.

The second component is a downstream test of subgroup modeling. We will train pooled, group-decoupled and jurisdiction-decoupled predictors on the proxy label. We then measure how much of each model's between-group difference in predicted risk lines up with the estimated $\delta_j$. We also test a correction: reweight or recalibrate training labels by the estimated recording effects, then refit.

The third component is a modest policy linkage. For each state in the NIBRS sample we will hand-code a small set of statutory and administrative features of the arrest process, such as arrest mandates for certain assault categories, and relate them to the $\tau$ estimates. With only about sixteen states this analysis is descriptive and low-powered, and we will present it as hypothesis generation.

One limitation is fixed by the data. NIBRS records only incidents known to police. We therefore identify the arrest-given-reported-incident stage of proxy generation, not the reporting stage, and every claim will be scoped accordingly.

Baselines

The naive reference point is a raw comparison of arrest rates by jurisdiction and group, with no adjustment for case mix. This is what aggregate policy statistics report. The file shows it can mislead: Tennessee simple-assault data show a 20% higher arrest rate for white offenders, a gap that disappears within cities, according to On the Validity of Arrest as a Proxy for Offense.

The strongest existing method is the Fogliato-style analysis: logistic regression of arrest on race, case features and jurisdiction fixed effects, run on the same incidents with the same feature set. Its reported reference values are the roughly 4% within-city racial gap and the Memphis-versus-Nashville arrest rates. Beating it does not mean a different point estimate on real data, where truth is unknown. It means three things on semi-synthetic data, where truth is known: lower error and nominal coverage for $\tau_{j,g}$; honest refusal to estimate where overlap fails; and stable estimates under misspecification. On real NIBRS data, our estimator should reproduce its within-city gap as a consistency check.

We also run controlled evaluation from Understanding challenges to the interpretation of disaggregated evaluations, applied across jurisdictions with the same reweighting budget. No reported numerical result for it was found on Lacuna; the file reports only its qualitative ACSIncome versus ACSPublicCoverage contrast. It counts as beaten if it labels a shift "outcome shift" that our method correctly attributes to recording.

For downstream modeling, the baselines are a pooled gradient-boosted model with group and jurisdiction indicators and their interactions, and a decoupled per-group model. TREE01 itself, which reports a 30.8% maximum gain on apnea, is too expensive to solve exactly at NIBRS scale, so we use per-group boosted trees as its scalable stand-in and run TREE01 only on small subsamples. All models get the same hyperparameter search budget and the same folds.

An outcome-regression-only estimator, without propensity weights, isolates whether the balancing ingredient matters. A version that ignores $G$ isolates the subgroup ingredient.

Why not just…?

Why not just fit a regression with jurisdiction fixed effects and race interactions? This is the real threat. A flexible outcome model may recover much of what balancing recovers. Our view is that a fixed-effects coefficient silently extrapolates across case mixes that do not overlap. Jurisdictions with 89% versus 57% Black offender shares are exactly where overlap is thin, and the regression gives no warning. The outcome-regression-only ablation settles this. If it matches the doubly robust estimator on bias and coverage in the semi-synthetic study, we will say the contribution is the estimand, the overlap reporting and the downstream finding, not the weighting.

Why not just collect gold labels and use DSL or the computational-phenotype estimator? Where gold labels exist, those estimators are better tools, and in the semi-synthetic study we run DSL with a small gold sample per state as a near-oracle comparison. For offending, gold labels cannot be collected at any price. The second problem is that both methods assume the feature–label relationship is shared between labeled and unlabeled data. Jurisdiction-varying recording breaks that assumption, and we will show it by having DSL pool gold labels across states that record differently.

Why not just impose fairness constraints on a pooled model? As Fair prediction with disparate impact argues, equalizing error rates against a tainted label only aligns the model with how enforcement varies. We include an equalized-odds-constrained model in the downstream comparison and expect it to improve parity on the proxy without improving construct-level calibration.

Why not just train subgroup-specific models, since they help when outcome relationships differ? That is the hypothesis under test, not an alternative to it. If decoupled models also improve construct-level accuracy in settings where recording is group-specific, our central warning is wrong.

Proposed Experiments and Decisive Checks

The first experiment is semi-synthetic and uses ACS PUMS public-coverage data across states. The true coverage label is the construct. A proxy label is generated by state- and group-specific recording probabilities, which depend partly on synthetic "policy features." The truth for $\tau_{j,g}$ is therefore known. We vary three things: overlap, whether recording depends on group, and the strength of an injected unmeasured severity variable.

Metrics for this experiment are bias, RMSE and 95% interval coverage for $\tau_{j,g}$ and $\delta_j$, plus rank correlation of recovered state orderings. For the downstream models, we report AUC and calibration error measured against the construct as well as the proxy.

The decisive check comes from comparing two conditions. When recording depends on group, decoupled models should gain on proxy AUC and lose on construct calibration relative to pooled models, and the label-corrected models should close that loss. When recording is uniform, the pattern should vanish. If it does not vanish, the effect comes from sample size, not proxy inheritance.

The second experiment uses NIBRS. We estimate $\tau$ between agencies within Tennessee first, which lets us check against the published Memphis and Nashville figures, and then across all sixteen states, separately by crime type. One falsification test is a placebo: split a single agency's incidents at random into two pseudo-jurisdictions; the estimates should be zero within their intervals. A second asks whether $\delta_j$ stays near the reported ~4% within-city level while $\tau_{j,g}$ varies widely across agencies. That pattern would mean clearance differs by agency more than by race within agency. If balancing makes the between-agency differences vanish, the clearance explanation fails and we report it.

The final step uses the NIBRS estimates to train arrest-based risk models under each design. We report the share of each model's predicted between-group risk gap that the estimated recording effects account for. This gives a quantitative, per-jurisdiction answer to a question current policy maps leave to provider discretion.