Is a Finding Failing to Transfer, or Is the Linkage Failing?

Studies built on linked government records often report that a result found in one region does not hold in another. A mortality gap, a risk model's accuracy, or a program effect shifts across states or counties. Some of that shift is real. Some of it may come from the record-matching step itself. Each region's registries carry their own typing errors, naming conventions, and gaps in identifiers, so a matching pipeline tuned in one region silently misses or invents links in another. Missed links in a mortality registry look exactly like lower mortality. This proposal builds a way to estimate, in a new region and without hand-labelled matches, how many links the pipeline gets wrong for each population subgroup. It then subtracts that error from the cross-region comparison, so analysts can tell genuine non-transfer apart from linkage artifact.

The approach exploits a resource that almost every large linkage already contains and that current pipelines use only as a shortcut. Some records in each region carry a reliable unique identifier, such as a national ID or health-system number, and those records can be matched exactly. Everyone else must be matched probabilistically, by comparing names, birth dates and places. The ID-bearing records therefore provide free ground truth inside every region. The obstacle is that they are not a random sample: people with recorded IDs tend to be better served by the state, and their records are cleaner. The core move is to treat ID-bearing records as labels collected under a known selection mechanism and to reweight them toward the ID-less population. This yields region- and subgroup-specific estimates of false-match and missed-link rates, which then feed into the downstream estimates whose transferability is in question.

Core Claim

Two claims are tested, and either can fail on its own. The first is about measurement. In a target region, selection-reweighted anchor records estimate subgroup linkage error better than three alternatives: carrying over the source region's similarity threshold, refitting an unsupervised match model per region, or a small clerical-review sample of comparable cost. Here "subgroup linkage error" means the false-match and missed-link rates within each subgroup. The second claim is substantive. Once those error estimates are propagated into downstream analyses, a meaningful share of observed cross-region differences in effect estimates shrinks. "Meaningful" is defined below as a pre-registered fraction, not a vague threshold.

Grounding in Prior Work

The motivating evidence comes from the linkage side. On the Accuracy and Scalability of Probabilistic Data Linkage Over the Brazilian 114 Million Cohort built Atylmo, which combines privacy-preserving Bloom-filter encodings, rule-based blocking (blocking means grouping records so only plausible pairs are compared), exact matching on stable fields, and Sørensen–Dice similarity thresholds on noisy strings. It reports true match rates of 93–97%. Its authors also note that the best Dice threshold varied with the database pair and with each state's data-entry quality, and they leave threshold calibration without ground truth unresolved. That open problem is the starting point here.

Learning Blocking Schemes for Record Linkage learns blocking rules from labelled pairs and evaluates them only on the distribution they were learned from. We reuse its metrics, Pairs Completeness and Reduction Ratio, and ask how they hold up across regions.

Performance Bounds for Graphical Record Linkage constrains the design. It proves that once per-field distortion in categorical data exceeds roughly 5%, records of distinct people become indistinguishable. No recalibration can recover that subgroup. We therefore use the estimated per-field distortion as a gate: past the cliff, the method reports "not identifiable" rather than a corrected number. That bound assumes one distortion rate for all records. We estimate rates per subgroup instead, which is the setting the bound does not cover.

Febrl supplies standard Fellegi–Sunter machinery. Fellegi–Sunter is the classical model that scores a record pair by how likely each field's agreement is among true matches versus non-matches. Febrl also provides standardisation and comparison functions, which we reuse as the probabilistic matcher.

On the transferability side, the applied linked-data studies supply findings whose portability is openly questioned. Socioeconomic and temporal heterogeneity in SARS-CoV-2 exposure and disease in England reports deprivation incidence rate ratios of 1.80 for hospitalisation and 1.64 for death, using deterministic linkage on hashed NHS numbers. The Hello Baby predictive risk model reports AUC 0.93 on Allegheny County's integrated warehouse. Its authors explicitly do not test transfer to other jurisdictions or sensitivity to linkage error across subgroups. X59 redistribution using linked data shows that whether linkage changes a policy conclusion depends on who sits in the ambiguous records. That is the same subgroup-dependence we expect for linkage error. These studies motivate the question but do not do what we propose. None of them estimates linkage error by subgroup in a target region, and none separates linkage artifact from real heterogeneity.

The broader generalizability literature offers an analogy rather than a method. Research on Urban Road Design Method in South China Based on Climate Zoning found that interpolation across mountainous terrain failed until a measured covariate, elevation, was brought in through co-kriging. Our counterpart is to condition linkage-error estimates on measured data-quality covariates such as region, registry source and record age, rather than assuming error is uniform.

Research Program

The system has three stages.

Stage 1: per-region probabilistic linkage. In each region, run the probabilistic matcher on all records, including the ID-bearing ones, as if their identifiers were hidden.

Stage 2: reweighted error estimation. For each ID-bearing pair, the exact-ID match is the label, so false-match and missed-link indicators are observed. Fit a model of having an ID given covariates $x$ (age, sex, region, area deprivation, registry source). Reweight anchor records by the odds of lacking an ID. For a similarity threshold $t$, the target-population missed-link rate is then estimated as

$$\hat{e}(t, g) = \frac{\sum_{i \in A_g} w_i\, m_i(t)}{\sum_{i \in A_g} w_i}, \qquad w_i = \frac{1-\pi(x_i)}{\pi(x_i)},$$

where $A_g$ is the set of anchor records in subgroup $g$, $m_i(t)$ indicates that the probabilistic matcher missed record $i$'s true link at threshold $t$, and $\pi(x_i)$ is the estimated probability that a record with covariates $x_i$ carries an ID. The false-match rate is estimated the same way. Per-field distortion is estimated by comparing field values within anchor pairs, and this feeds the 5% identifiability gate.

Stage 3: propagation into downstream estimates. Use the subgroup error rates to correct outcome counts by standard misclassification adjustment. Use the bootstrap over anchors to carry estimation uncertainty forward, so that cross-region confidence intervals widen honestly. The downstream quantity is whatever finding is being transported: a rate ratio, a risk model's AUC and top-5% precision, or a program effect.

The method's main assumption is that, conditional on $x$, anchor records and ID-less records suffer the same field corruption. This is not testable from the data alone. We probe it in two ways. First, in synthetic data we deliberately violate it. Second, on real data we check whether the correction's accuracy degrades as $\pi(x)$ falls.

Baselines

The naive reference point is source-threshold transfer. The Dice threshold and blocking rules are tuned in the region with the best ground truth and then applied unchanged everywhere, which is how pipelines are typically exported. It sets the "do nothing" level of bias.

The strongest competitor is per-region unsupervised Fellegi–Sunter refitting. This means fitting the match/non-match agreement probabilities by expectation-maximisation, a standard iterative fit for hidden labels, separately in each region and deriving thresholds and implied error rates from the fitted model. It uses Febrl's implementation. It is strong because it already adapts to local data quality and needs no labels, so it directly attacks the Atylmo threshold-instability problem. We give it the same blocked candidate pairs, the same comparison features, and the same tuning budget for initialisation and convergence restarts.

The third baseline is clerical review. A random sample of candidate pairs per region is hand-labelled, with the sample size set so its cost matches our anchor-processing effort. Error is estimated from that sample. In synthetic experiments, review labels are drawn from ground truth with a realistic reviewer error rate.

The key ablation is anchors without reweighting, which isolates whether selection correction is what does the work.

For the strongest existing linkage numbers, the file reports a 93–97% true match rate for Atylmo on the Brazilian cohort (Brazilian 114 Million Cohort). It also reports, for blocking on synthetic census data, 99.85% Pairs Completeness against 99.16% for an expert's hand rules (Learning Blocking Schemes). No reported result found on Lacuna for cross-region transfer of linkage thresholds or for subgroup-level linkage-error estimation, so the baseline numbers on that question will be produced here.

Beating a baseline means lower mean absolute error in estimated subgroup missed-link rate, averaged over held-out regions and subgroups. It also means bootstrap intervals with coverage close to nominal, in at least the subgroups where $\pi(x)$ is low. Those are the subgroups where errors bias findings most.

Why not just…?

Why not just refit Fellegi–Sunter in each region? This is the real threat. Unsupervised refitting does recalibrate thresholds, and if its implied subgroup error rates prove as accurate as the anchor estimates, that is the answer and we will report it. Our reason to expect a gap is structural rather than cited. The model assumes fields disagree independently, and in administrative data name and mother's-name errors are correlated. It also yields error rates for the pooled population rather than by subgroup unless it is refit per subgroup, and true matches are rare within subgroups. The synthetic experiments vary field correlation and subgroup size precisely to locate where, if anywhere, refitting breaks.

Why not just hand-review a sample in each region? Clerical review is unbiased and costs little to run once, but at fixed cost it puts few labels into small, high-error subgroups. Those are the subgroups that drive differential bias. The cost-matched comparison tests this directly. If review matches anchor accuracy, the value of the proposal reduces to cost savings, and we will say so.

Why not just use deterministic matching on IDs only? The Brazilian cohort paper shows that IDs are missing or corrupted for large, often vulnerable segments. Restricting to ID-bearing records changes the population, which is itself a transferability failure. Our method uses those records as labels instead of as the analytic sample.

Proposed Experiments

Synthetic regions. Generate a population with Febrl-style field corruption. Split it into regions with different per-field distortion levels, name frequency distributions, correlated field errors, and ID coverage that depends on covariates. Plant a downstream association, a mortality rate ratio between two exposure groups. In some regions its true value is identical across regions; in others it truly differs. Primary metrics are subgroup error-rate accuracy, interval coverage, and the fraction of observed cross-region difference attributable to linkage. Because the truth is known, this is the decisive test of whether the method separates artifact from heterogeneity. It should shrink spurious differences and leave genuine ones intact.

Masked-ID real data. Use systems with near-universal identifiers, such as SAIL in Wales or the NHS-number linkage used in the England study. Hide identifiers for a covariate-dependent subset, run probabilistic linkage, and score against the hidden truth. Re-estimate a published-style quantity, for example deprivation rate ratios by region of England or injury–education associations, under each method. The decisive check is whether corrected regional estimates move toward the true-ID estimates while naive ones do not.

Brazilian cohort, if access is granted. Compare Bolsa Família–mortality associations across states, with and without correction. Report where the identifiability gate fires.

The study does not succeed merely by beating source-threshold transfer, since any adaptive method should. It succeeds if reweighted anchors beat per-region refitting on low-coverage subgroups, and if the share of cross-region differences that disappear after correction is large enough to change a policy-relevant conclusion in at least one real comparison.