Abstract
Clustering analysis of functional data, which comprises observations that evolve continuously over time or space, has gained increasing attention across various scientific disciplines. Practical applications often involve functional data that are contaminated with measurement errors arising from imprecise instruments, sampling errors, or other sources. These errors can significantly distort the inherent data structure, resulting in erroneous clustering outcomes. In this paper, we propose a simulation-based approach designed to mitigate the impact of measurement errors. Our proposed method estimates the distribution of functional measurement errors through repeated measurements. Subsequently, the clustering algorithm is applied to simulated data generated from the conditional distribution of the unobserved true functional data given the observed contaminated functional data, accounting for the adjustments made to rectify measurement errors. We illustrate through simulations show that the proposed method has improved numerical performance than the naive methods that neglect such errors. Our proposed method was applied to a childhood obesity study, giving more reliable clustering results.
Introduction
In fields like medicine and engineering, researchers often work with functional data—observations that represent continuous processes over time or space. Examples include energy expenditure measured minute-by-minute or heart rate monitoring over a 24-hour period. A primary goal in analyzing such data is to identify subgroups of individuals who share similar patterns. However, data collected via sensors and wearable devices is frequently contaminated by measurement errors. Standard clustering algorithms typically treat this noise as part of the signal, leading to "over-clustering." This phenomenon creates spurious subgroups or "singletons" (clusters containing only one individual) because noise makes a subject's data appear unique. This paper introduces a simulation-based framework to "de-noise" the clustering process and reveal true underlying structures.
Methodology
The proposed approach builds upon a pairwise-grouping penalization method, incorporating a novel simulation-based correction:
- Functional Approximation: Discrete, noisy data points are converted into smooth mathematical functions using B-splines.
- Error Estimation via Repetition: The method utilizes repeated measurements (e.g., multiple days of data for the same subject). By comparing these repetitions within identified subgroups, the algorithm estimates the variance and covariance of the measurement error.
- Simulation-Based Correction: This is the core innovation. Instead of clustering raw noisy data, the method calculates the conditional distribution of the true unobserved function given the observed noisy data. "Pseudo-functional" data is then simulated from this distribution, which exhibits significantly reduced variability.
- Iterative Refinement: The process is iterative. It starts with an initial "naive" cluster assignment, uses those clusters to estimate the error distribution, re-simulates the data, and re-clusters until assignments stabilize, monitored via the Bayesian Information Criterion (BIC).
- Automatic Cluster Discovery: Unlike K-means, the pairwise penalty approach automatically determines the optimal number of clusters ($K$).
Simulation Results
The authors evaluated the method using simulated datasets with $n=40$ or $80$ subjects split into two groups. Scenarios varied by group distinctness ("Far" vs. "Close") and noise levels ($\sigma=2$ and $3$).
As shown in the simulation results table, the proposed method ($Y_i^*(t)$) consistently outperformed the naive method ($\bar{W}_i(t)$). In the "Far" scenario with high noise ($\sigma=3$), the naive method identified an average of 7.64 clusters, whereas the proposed method identified 2.54, much closer to the ground truth of 2. Furthermore, the proposed method yielded higher Rand, Jaccard, and adjusted Rand (aRand) indices across almost all scenarios, indicating superior alignment with the true data structure.
Real-World Application: Childhood Obesity Study
The method was applied to data from 45 third-grade students, tracking Daily Energy Expenditure (DEE) over three days using accelerometers. The naive method identified 10 clusters, many of which were singletons. In contrast, the proposed simulation-based method consolidated these into 5 distinct, interpretable groups. This reduction in "noise-driven" clusters provides a more reliable basis for clinical interventions and understanding physical activity patterns in children.
Conclusion
Accounting for measurement error is critical in functional data clustering. By using repeated measurements and a simulation-based correction, the proposed method effectively mitigates the "singleton" problem and provides more accurate cluster recovery. While the method requires repeated measurements and carries a higher computational cost than naive approaches, its ability to recover true underlying structures in noisy environments makes it a valuable tool for functional data analysis.
Summary updated