Many risk tools in child welfare, health care and criminal justice are trained on a label with two defects at once. The label is recorded only for people whom an earlier decision let through, such as families investigated, patients admitted or defendants released. This is the selective labels problem. And where the label is recorded, it measures contact with an institution rather than the thing we care about: child removal instead of maltreatment, cost instead of illness, arrest instead of offending. This is proxy label bias. Existing corrections handle one defect and assume the other away. Methods that fill in labels for rejected people copy the proxy's bias onto them. Methods that correct noisy labels assume every person has a label.
Our idea is to use record linkage across agencies to obtain a second, rarer outcome for everyone, whether or not they passed the decision. Examples are a death record, a coroner's finding, or a fatality or near-fatality review. We treat this linked outcome as an anchor: a signal that is rare and incompletely linked, but produced outside the institution whose decisions censor the main label. The anchor does not replace the proxy. It calibrates the proxy, separately for each demographic subgroup. The abundant proxy then supplies statistical power, and the anchor supplies validity. We test when this combination beats simply training on the anchor, and when subgroup-specific calibration is worth its variance.
Core Claim
When a linked outcome exists that meets two conditions, the per-group rates at which the proxy misses true cases can be identified even under selective labels. The first condition is that the anchor is recorded independently of the gating decision, given the true state. The second is that a positive anchor almost never occurs without the true condition. The per-group false-alarm rates of the proxy and the overall risk level are then identified up to one interpretable quantity: how often the anchor fires when the condition is present, after accounting for how often linkage succeeds in that group. A model fit to all three signals jointly (gate, proxy, anchor), with group-specific measurement parameters shrunk toward a shared value, should do two things. It should rank people by latent need better than a proxy-trained model. In small subgroups it should be more efficient than a model trained on the anchor alone. We expect the gain to be largest exactly where current tools fail: groups that are both rarely selected and poorly linked.
This addresses a combination the research file flags as unformulated, namely simultaneous proxy bias and selective censoring. It differs from the closest work in specific ways.
On Group Sufficiency Under Label Bias corrects group-dependent label noise with known transition matrices and the CMI-REG regularizer. It needs an observed proxy for every person, and it takes the noise rates as given rather than estimating them. We supply an estimate of those rates from linked data under censoring.
Long-term Fairness with Selective Labels (SELLF) fills in outcomes for rejected people. Residual Unfairness in Fair Machine Learning from Prejudiced Data reweights toward an unlabeled census population. Both treat the observed label as ground truth, which is the assumption we drop.
The Hello Baby model, described in Using a Predictive Risk Model to Prioritize Families for Prevention Services, comes closest in spirit. It checked its removal-trained score against infant mortality and maltreatment fatalities after training, reporting a relative risk of 5.54 in the top 5% of scores. We move that check inside the estimator and make it group-specific.
Model
Let $Y^*$ be the latent binary state, $A$ the subgroup, $X$ the features, and $D$ the gate. The proxy $Y$ is seen only when $D=1$. Let $R$ indicate successful linkage, with linkage rate $\rho_a(x)=P(R=1\mid X=x,A=a)$, and let $L$ be the anchor seen when $R=1$. We make three assumptions.
The first is that $L$ and $Y$ are conditionally independent given $Y^*,X,A$.
The second is that the anchor almost never fires without the condition: $P(L=1\mid Y^*=0)\approx 0$.
The third is that the anchor's sensitivity $s_a=P(L=1\mid Y^*=1,A=a)$ does not depend on $D$. This is the crux, and it can fail, for example when removal prevents a fatality. We handle that below.
Under the first two assumptions, among linked, selected people with a positive anchor, the proxy's hit rate directly estimates its sensitivity:
$$P(Y=1\mid L=1, R=1, D=1, A=a)=P(Y=1\mid Y^*=1, D=1, A=a).$$
The anchor's own rate in the full linked population, whether selected or not, gives
$$P(L=1\mid R=1, X, A=a)=s_a\,P(Y^*=1\mid X, A=a),$$
which identifies risk across the whole population up to the scalar $s_a$.
Fixing $s_a$ by domain knowledge, or reporting results across a range of values, then identifies the proxy's false-alarm rate and latent risk for each group. Ranking within a group, which is what top-$k$ allocation needs, is identified without knowing $s_a$. Comparisons across groups require only the ratio $s_a/s_b$. That ratio becomes the single sensitivity parameter we report.
Research Program
We will build a latent-variable learner. Its parts are a group-indexed risk model $f_a(x)\approx P(Y^*=1\mid x,a)$, gate and linkage models, and measurement parameters $(T_a, s_a)$. The group parameters are written as a shared component plus a penalized group deviation. This partial pooling is the subgroup-specific element. It lets a small group's miss rate differ from the majority's only as far as the data support, which avoids the sample starvation that Fairness without Harm: Decoupled Classifiers with Preference Guarantees identifies for fully separate models.
The learner is fit by maximizing the observed-data likelihood of $(D,\ Y\cdot D,\ R,\ L\cdot R)$ with cross-fitting. We borrow the logic of Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift, which shows that cheap proxies improve efficiency mainly through the unlabeled pool, so the proxy enters as an auxiliary signal. We also derive the efficiency gain over anchor-only training as a function of anchor prevalence, linkage rate and proxy sensitivity. The gain should grow roughly as the ratio of the proxy's effective information to the anchor's, the standard two-signal variance argument.
We treat violation of the third assumption as a bounded sensitivity parameter: the anchor's sensitivity may differ between selected and unselected people by a factor within a stated range. We report how rankings and group calibration degrade across that range.
Linkage failure that depends on the group enters through $\rho_a(x)$. It can be estimated from clerical-review audits where those exist. Otherwise it is treated as a second sensitivity parameter. We never assume linkage fails at random.
Baselines
The naive reference is a gradient-boosted classifier trained on the proxy among selected people only, which is how deployed tools like Hello Baby's LASSO are built. Hello Baby reports AUC 0.93 against its removal proxy. That figure is not comparable to our target, and matching it would not count as success.
The first strong competitor is anchor-only training in the style of Dissecting racial bias in an algorithm used to manage the health of populations. That paper retrained on a less-biased target (chronic conditions) and reported an 84% reduction in racial bias. Here the analogue is fitting $f$ directly to $L$ among linked people, with linkage weighting. It is the most serious threat, because it is valid under the same assumptions.
The second is SELLF-style imputation of outcomes for rejected people, run in its static single-step form. In its lending environment it reached true disparity 0.03 against an oracle's 0.05, per Long-term Fairness with Selective Labels.
The third is propensity reweighting to the full population, following Residual Unfairness in Fair Machine Learning from Prejudiced Data.
The fourth is CMI-REG given oracle transition matrices and given matrices estimated naively. CMI-REG reports sufficiency-gap reductions of up to 7.2% at unchanged AUROC in On Group Sufficiency Under Label Bias.
Two ablations isolate the key ingredients. One uses fully pooled measurement parameters, removing the subgroup component. The other uses fully separate per-group parameters, removing the shrinkage. All methods share features, cross-fitting splits, model family and the same hyperparameter budget of 50 trials, tuned on a validation fold scored against the anchor only, never against the truth.
Beating the baselines means lower group-wise calibration error and a lower sufficiency gap against the true state than every baseline in the smallest two subgroups, with no loss in overall top-5% capture of true cases. Against anchor-only training specifically, it means narrower intervals for group calibration at matched bias.
Why not just…?
Why not just train on the linked anchor? When the anchor is common and well linked, this may match our method. Our own theory predicts that, and we will report it. The case for the method rests on rare anchors such as fatalities, and on small, poorly linked groups, where anchor-only training has few positives per group. Experiment 1 varies anchor prevalence and linkage rate to map where the two cross over. If anchor-only training wins everywhere, that is a publishable negative result, and we would recommend it.
Why not a pooled model with group indicators? Indicators let the risk surface shift by group, but they do not let the proxy's error rates differ by group. That difference is exactly where cost-versus-need bias sits, as Dissecting racial bias in an algorithm used to manage the health of populations shows. The pooled-measurement ablation is this alternative with the anchor added, so it tests the point directly.
Why not fill in or reweight the censored labels? Both SELLF and propensity reweighting reproduce the proxy for rejected people. When the proxy's miss rate differs by group, they carry that difference into the population. Experiment 1 injects precisely this difference, so the size of the carried-over bias will be measured rather than asserted.
Why not CMI-REG with guessed noise rates? CMI-REG needs the transition matrices and a label for every person. We run it both with oracle matrices and with our estimated ones, which shows whether our contribution is simply supplying the right inputs to CMI-REG. That combination is a legitimate outcome of this work.
Proposed Experiments
Experiment 1: semi-synthetic benchmarks with known truth. On FICO, COMPAS and Adult we will inject a latent $Y^*$, a gate that depends on the group, proxy miss and false-alarm rates that depend on the group, and a rare anchor with linkage rates of, for example, 95% for the majority and 70% for the minority. We sweep anchor prevalence, linkage gap, and violation of the third assumption.
Experiment 2: real linked outcomes with a held-out anchor. We use MIMIC-III linked to Social Security death records, which report 11.8% one-year mortality among 30-day survivors in One-year mortality after recovery from critical illness. The facility-recorded outcome serves as the gated proxy, observed only for patients who return. We hide the death-record link for most patients, making it a sparse anchor, and keep the full link as the truth for evaluation. Groups are defined by sex and insurance.
Experiment 3 (conditional on data access): Allegheny County. On the Allegheny warehouse, removal is the proxy and fatality or near-fatality and infant mortality are anchors. We would replicate the external check from Using a Predictive Risk Model to Prioritize Families for Prevention Services by group.
The decisive checks come from Experiments 1 and 2. First, the joint model must beat anchor-only training on group calibration in the regime where anchors are rare. Second, it must beat pooled measurement whenever the injected proxy miss rates differ by group, and it must not beat pooled measurement when they do not; failing that placebo would indicate overfitting. Third, its advantage must survive the reported range of anchor sensitivity ratios.
References
- On Group Sufficiency Under Label Bias
- Long-term Fairness with Selective Labels
- Residual Unfairness in Fair Machine Learning from Prejudiced Data
- Using a Predictive Risk Model to Prioritize Families for Prevention Services: The Hello Baby Program in Allegheny County, PA
- Fairness without Harm: Decoupled Classifiers with Preference Guarantees
- Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift
- Dissecting racial bias in an algorithm used to manage the health of populations
- One-year mortality after recovery from critical illness: A retrospective cohort study