In fields ranging from clinical medicine to developmental biology, researchers rarely have the luxury of continuous monitoring. Instead, they often work with "fragmentary" data: a handful of blood pressure readings, a few growth measurements, or irregular voting records. Traditional time-series methods struggle here because they expect dense, regularly spaced observations. Sparse Functional Data Analysis (sFDA) solves this by treating these scattered points not as a sequence of discrete numbers, but as sparse realizations of an underlying continuous stochastic process.
Borrowing Strength Across the Population
The core intuition behind sFDA is that while an individual subject’s data might be too sparse to reconstruct their specific curve, the population as a whole provides a blueprint of how those curves typically behave. By pooling observations from all subjects, researchers can estimate a global mean function and a shared covariance structure.
This "borrowing of strength" allows for the recovery of smooth underlying trends even from subjects with as few as two or three data points. As Jeng‐Min Chiou and Hans-Georg Müller describe in Review of Functional Data Analysis, this process leverages the shared covariance to impute missing segments of a curve, effectively filling in the gaps for each individual based on the patterns seen in the group.
Principal Analysis by Conditional Expectation
The most influential mechanism for this reconstruction is Principal Analysis by Conditional Expectation (PACE). Standard Functional Principal Component Analysis (FPCA) requires dense data to calculate the "scores" that define an individual's unique trajectory. PACE bypasses this requirement.
Instead of integrating over a full curve, PACE uses the conditional expectation of the principal component scores given the sparse observations. This allows researchers to identify the dominant "modes of variation" across the population—such as the "drift" in ideology among Supreme Court justices described by Xiner Zhou in The dynamics of ideology drift among U.S. Supreme Court justices—and then project a sparse individual onto those modes.
Smoothing and Imputation in Regression
When these sparse curves are used as predictors in a regression model, the challenge of noise and missingness is amplified. Frameworks like MISFIT, developed by Justin Petrovich and Carrie Daymont, use multiple imputation to handle this uncertainty. By reconstructing latent trajectories from the conditional distribution of FPCA scores, these models provide consistent parameter estimates even when medical records are highly irregular.
Similarly, in Functional Linear Regression for Partially Observed Functional Data, Yafei Wang and colleagues demonstrate that the relationship between a functional predictor and an outcome can be recovered by decomposing scores into observed and missing components. This ensures that even "fragmentary" curves—those observed only over a partial interval—can contribute meaningful information to the model.
Beyond Linear Trajectories
While much of the field relies on linear decompositions, recent work has pushed sFDA into more complex territories:
- Nonlinear Learning: Functional Nonlinear Learning (FunNoL) uses neural networks to capture hidden structures that traditional FPCA might miss, particularly when data is noisy.
- Manifold Data: Xiongtao Dai extended the PACE algorithm to Riemannian manifolds in Modeling Longitudinal Data on Riemannian Manifolds, allowing for the analysis of sparse longitudinal data that exists on curved surfaces, such as shapes or compositions.
- Derivatives: Methods like Derivative PCA allow researchers to study the rate of change (velocity) of a process directly from sparse points, which is often more informative for biological growth or physical dynamics than the raw values themselves.