A new method for estimating under-recruitment of a patient registry: a case study with the Ohio Registry of Amyotrophic Lateral Sclerosis

Meifang Li, Xun Shi, Jiang Gui, Chao Song, Angeline S. Andrew, Erik P. Pioro, Elijah W. Stommel, Maeve Tischbein, Walter G. BradleyScientific 2022 · DOI

Abstract

We developed a disease registry to collect all incident amyotrophic lateral sclerosis (ALS) cases diagnosed during 2016-2018 in Ohio. Due to incomplete case ascertainment and limitations of the traditional capture-recapture method, we proposed a new method to estimate the number of cases not recruited by the Registry and their spatial distribution. Specifically, we employed three statistical methods to identify reference counties with normal case-population relationships to build a Poisson regression model for estimating case counts in target counties that potentially have unrecruited cases. Then, we conducted spatial smoothing to adjust outliers locally. We validated the estimates with ALS mortality data. We estimated that 119 total cases (95% CI [109, 130]) were not recruited, including 36 females (95% CI [31, 41]) and 83 males (95% CI [74, 99]), and were distributed unevenly across the state. For target counties, including estimated unrecruited cases increased the correlation between the case count and mortality count from r = 0.8494 to 0.9585 for the total, from 0.7573 to 0.8270 for females, and from 0.6862 to 0.9292 for males. The advantage of this method in the spatial perspective makes it an alternative to capture-recapture for estimating cases missed by disease registries.

Introduction: The Challenge of Incomplete Registries

Patient registries are vital for understanding rare diseases like Amyotrophic Lateral Sclerosis (ALS), helping researchers identify geographic "hotspots" and potential environmental risk factors. However, these registries are rarely complete. In the United States, reporting is often voluntary, and patients may be diagnosed at out-of-state centers or facilities that do not participate in the registry.

The traditional gold standard for estimating missing patients is the capture-recapture method. While useful for estimating total numbers, it has significant drawbacks: it requires multiple independent data sources and is non-spatial. It can estimate how many people are missing across a large area, but not where they live. Without accurate geographic data, researchers cannot reliably study the links between environmental exposures and disease prevalence.

Methodology: A Two-Phase Spatial Estimation Approach

To address these limitations, this study proposes a novel two-phase approach that relies on local data to estimate the spatial distribution of unrecruited cases.

Phase 1: Statistical Modeling via "Reference Counties" The researchers first identified "reference counties"—areas where the relationship between population and reported ALS cases appeared stable and representative. They used three statistical filters to select these counties: z-scores, visual inspection of ranked incidence rates, and Jenk’s natural breaks. A Poisson regression model was then built using these reference counties to predict the "expected" number of cases in "target counties" (those suspected of under-recruitment).

Identification of reference counties using ranked incidence rates, differences, and ratios for females (a) and males (b).

Phase 2: Spatial Adjustment To ensure the model did not overlook real biological clusters, the team applied spatial smoothing. Using Local Moran’s I analysis, they identified local anomalies, such as low-incidence counties surrounded by high-incidence neighbors. A locally-weighted average was then used to adjust these estimates, ensuring the final map accounted for regional trends while maintaining geographic integrity.

Data Validation

The study utilized three primary datasets:

  • Ohio ALS Registry (2016–2018): Recruited incident ALS cases.
  • NCHS Population Data: County-level population data for Ohio (2016).
  • Ohio Bureau of Vital Statistics (Mortality Data): Used as an independent "ground truth" to validate estimates. Researchers used a 4-year median survival time to estimate diagnosis locations for deceased patients.

Key Results: Impact on Disease Mapping

The model estimated that 119 cases (95% CI [109, 130]) were missed by the registry over the study period, accounting for approximately 29.6% of total estimated ALS cases in Ohio.

Comparison of ALS incidence rates in Ohio before (a) and after (b) accounting for estimated unrecruited cases.

A significant finding was the sex disparity in reporting: unrecruited cases were notably higher in males (34.87%) than females (21.95%). This included 44 veterans diagnosed within the Veterans Affairs (VA) system who were not captured by the initial registry.

The inclusion of these estimated missing cases dramatically improved the accuracy of the disease map. In target counties, the correlation ($r$) between the registry's case counts and independent mortality data improved from 0.8494 to 0.9585 for total cases. For males, the improvement was even more stark, jumping from a weak 0.6862 to a robust 0.9292.

Conclusion and Future Directions

While the method successfully "recovered" the underlying distribution of the disease, some limitations remain. The "small number problem" inherent in rare diseases can lead to unstable incidence rates in low-population counties. Additionally, the accuracy of the model depends on the selection of reference counties and the assumption that patients do not move between diagnosis and death.

Despite these hurdles, this spatial statistical method offers a powerful alternative to traditional capture-recapture for any disease registry struggling with incomplete recruitment, providing the high-quality geographic data necessary for advanced epidemiological research.

Summary updated

Abstract

We developed a disease registry to collect all incident amyotrophic lateral sclerosis (ALS) cases diagnosed during 2016-2018 in Ohio. Due to incomplete case ascertainment and limitations of the traditional capture-recapture method, we proposed a new method to estimate the number of cases not recruited by the Registry and their spatial distribution. Specifically, we employed three statistical methods to identify reference counties with normal case-population relationships to build a Poisson regression model for estimating case counts in target counties that potentially have unrecruited cases. Then, we conducted spatial smoothing to adjust outliers locally. We validated the estimates with ALS mortality data. We estimated that 119 total cases (95% CI [109, 130]) were not recruited, including 36 females (95% CI [31, 41]) and 83 males (95% CI [74, 99]), and were distributed unevenly across the state. For target counties, including estimated unrecruited cases increased the correlation between the case count and mortality count from r = 0.8494 to 0.9585 for the total, from 0.7573 to 0.8270 for females, and from 0.6862 to 0.9292 for males. The advantage of this method in the spatial perspective makes it an alternative to capture-recapture for estimating cases missed by disease registries.
BibTeX
@inproceedings{li2022new,
  author = {Meifang Li and Xun Shi and Jiang Gui and Chao Song and Angeline S. Andrew and Erik P. Pioro and Elijah W. Stommel and Maeve Tischbein and Walter G. Bradley},
  title = {{A new method for estimating under-recruitment of a patient registry: a case study with the Ohio Registry of Amyotrophic Lateral Sclerosis}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-18944-9},
}
Figure