Highly Irregular Functional Generalized Linear Regression with Electronic Health Records

Justin Petrovich, Matthew Reimherr, Carrie DaymontJournal of the Royal Statistical Society Series C (Applied Statistics) 2022 · DOI · arXiv

Abstract

This work presents a new approach, called Multiple Imputation of Sparsely-sampled Functions at Irregular Times (MISFIT), for fitting generalized functional linear regression models with sparsely and irregularly sampled data. Current methods do not allow for consistent estimation unless one assumes that the number of observed points per curve grows sufficiently quickly with the sample size. In contrast, MISFIT is based on a multiple imputation framework, which, as we demonstrate empirically, has the potential to produce consistent estimates without such an assumption. Just as importantly, it propagates the uncertainty of not having completely observed curves, allowing for a more accurate assessment of the uncertainty of parameter estimates, something that most methods currently cannot accomplish. This work is motivated by a longitudinal study on macrocephaly, or atypically large head size, in which electronic medical records allow for the collection of a great deal of data. However, the sampling is highly variable from child to child. Using MISFIT we are able to clearly demonstrate that the development of pathologic conditions related to macrocephaly is associated with both the overall head circumference of the children as well as the velocity of their head growth.

Introduction

Functional Data Analysis (FDA) is a powerful tool for analyzing data that changes over time, such as growth curves or heart rates. However, most FDA techniques assume "dense" data—meaning we have many observations for every subject at the same time intervals. In the real world, especially with Electronic Health Records (EHR), data is "highly irregular." One child might have their head circumference measured at birth and six months; another might have five measurements in the first year.

Current gold-standard methods, such as PACE, often struggle with this irregularity because they perform "mean imputation"—filling in gaps with an average curve—while ignoring the outcome variable (e.g., whether the child actually has a medical condition). This leads to two major issues:

  1. Bias: Ignoring the outcome during imputation can lead to "attenuation bias," underestimating the true relationship between the curve and the outcome.
  2. Overconfidence: Mean imputation treats the "guessed" data as certain, leading to artificially small standard errors and misleadingly significant p-values.

Methods: The MISFIT Framework

The authors propose MISFIT (Multiple Imputation of Sparsely-sampled Functions at Irregular Times), which treats the sparse functional data problem as a missing data problem. Instead of filling in gaps with a single "best guess," MISFIT uses Multiple Imputation:

  • Outcome-Dependent Imputation: It estimates the conditional distribution of the missing curve segments given both the observed points and the outcome variable ($Y$).
  • Multiple Draws: It draws $K$ different plausible versions of the full curves (or their principal component scores).
  • Rubin’s Rules: It fits the regression model to each of the $K$ datasets and combines the results. This allows the model to "propagate uncertainty"—if the data is very sparse, the $K$ versions will vary wildly, resulting in wider, more honest confidence intervals.

Data and Sparsity

The primary motivation for this work is a macrocephaly EHR dataset containing 74,428 children from a large primary care network. The goal was to predict pathologic macrocephaly (atypically large head size caused by conditions like hydrocephalus or tumors).

Distribution of the number of observations per subject, highlighting the extreme sparsity of the EHR data.

As shown in the distribution above, the data is extremely sparse: the most common number of measurements per child is just one, and 98% of subjects had 10 or fewer measurements. Traditional methods that rely on dense sampling or simple mean imputation are ill-suited for such a distribution.

Key Results and Clinical Impact

In simulations, MISFIT consistently achieved the lowest Median Integrated Squared Error (MISE) compared to PACE and other benchmarks, especially when the signal was strong and data was sparse. Crucially, MISFIT maintained Type 1 error rates close to the nominal 0.05, whereas mean imputation methods saw rates as high as 0.704, indicating a high risk of "false discovery."

Head circumference trajectories for a subset of cases and controls, illustrating the variability in growth patterns.

When applied to the macrocephaly data, MISFIT revealed that it isn't just the absolute size of the head that predicts pathology, but the velocity of growth. Children whose head circumference grew rapidly relative to their height were at higher risk. Furthermore, MISFIT provided a more realistic assessment of uncertainty for clinicians, showing that traditional methods were dramatically underestimating the standard errors of their predictions.

Conclusion

MISFIT addresses a critical gap in functional data analysis for EHR data. By incorporating the outcome variable into the imputation process and using multiple draws to capture uncertainty, it provides more robust and honest statistical inferences. Future work includes extending the framework to Functional Generalized Additive Models (GAMs) and further exploring the minimax rates of convergence for highly irregular designs.

Summary updated

Abstract

Abstract This work presents a new approach, called Multiple Imputation of Sparsely-sampled Functions at Irregular Times (MISFIT), for fitting generalized functional linear regression models with sparsely and irregularly sampled data. Current methods do not allow for consistent estimation unless one assumes that the number of observed points per curve grows sufficiently quickly with the sample size. In contrast, MISFIT is based on a multiple imputation framework, which, as we demonstrate empirically, has the potential to produce consistent estimates without such an assumption. Just as importantly, it propagates the uncertainty of not having completely observed curves, allowing for a more accurate assessment of the uncertainty of parameter estimates, something that most methods currently cannot accomplish. This work is motivated by a longitudinal study on macrocephaly, or atypically large head size, in which electronic medical records allow for the collection of a great deal of data. However, the sampling is highly variable from child to child. Using MISFIT we are able to clearly demonstrate that the development of pathologic conditions related to macrocephaly is associated with both the overall head circumference of the children as well as the velocity of their head growth.
BibTeX
@article{petrovich2022highly,
  author = {Justin Petrovich and Matthew Reimherr and Carrie Daymont},
  title = {{Highly Irregular Functional Generalized Linear Regression with Electronic Health Records}},
  journal = {Journal of the Royal Statistical Society Series C (Applied Statistics)},
  year = {2022},
  doi = {10.1111/rssc.12556},
}
Figure