TL;DR
This paper introduces the "Stationarity Bias" in time-series imputation, showing how standard benchmarks unfairly favor simple methods by over-sampling stable periods. They propose a "Stratified Stress-Test" using Continuous Glucose Monitoring (CGM) data, revealing that while linear interpolation excels in stable states, deep learning models are crucial for accurately capturing critical transient events, a fact often masked by traditional metrics like RMSE.
What Problem Does This Paper Address?
Many real-world systems, like glucose regulation, power grids, or network traffic, spend most of their time in a stable, predictable state (homeostasis) but experience rare, critical "transient" events (e.g., meals, system failures). When evaluating time-series imputation methods (filling in missing data), standard benchmarks often use random masking. This approach disproportionately samples the dominant stable periods, making simple imputation methods (like linear interpolation) appear superior because they perform well in these easy conditions. This "Stationarity Bias" hides the fact that these simple methods fail dramatically during critical transient events, where more complex, deep learning models are actually needed for accurate and safe predictions. The paper aims to formalize this bias and propose a more robust evaluation framework.
Methods
The authors propose a "Stratified Stress-Test" that partitions time-series data into "Stationary" and "Transient" regimes. They use Continuous Glucose Monitoring (CGM) data as a testbed, leveraging meal and insulin inputs to precisely identify these regimes.
Key methodological contributions include:
- Formalizing the Stationarity Bias: Quantifying how random masking over-represents stable periods.
- Stratified Stress-Test: Evaluating models separately on stationary and transient intervals.
- The "RMSE Mirage": Demonstrating how low RMSE can mask significant morphological (shape) degradation during transients, introducing Dynamic Time Warping (DTW) as a complementary metric.
- Ecologically Valid Missingness: Simulating missing data based on real-world clinical trial distributions (DCLP3, DCLP5) rather than purely random masking.
- Regime-Conditional Model Selection: Proposing a strategy to use simpler models during stable periods and more complex ones during critical transients.
Datasets
The study utilizes the following datasets:
- Continuous Glucose Monitoring (CGM) data: Specifically, data from the DCLP3 and DCLP5 clinical trials were used to derive realistic missingness patterns.
- UVA/Padova simulation dataset: A widely accepted simulator for artificial pancreas technologies, used for generating synthetic data.
- TCR-Simulation Dataset: A specialized simulation dataset designed to mimic Temporal Control Reset events, often associated with hypoglycemia.
- PEDAP (Pediatric Artificial Pancreas) dataset: Used for training and testing, with both raw and processed versions.
The authors also introduce a method for generating "Ecologically Valid Missingness" by applying empirical missingness distributions from clinical trials to complete datasets, creating a more realistic evaluation scenario than standard random masking.
Key Insights and Ideas
- Stationary periods are easy, transient periods are hard (and critical): Simple methods like linear interpolation are perfectly adequate, even optimal, for stable, low-entropy periods. However, they fail catastrophically during dynamic, high-entropy transients.
- RMSE is a misleading metric for critical events: The "RMSE Mirage" highlights how low pointwise error can hide severe distortions in signal shape (morphology), which is crucial for safety-critical applications. DTW is a better measure for this.
- Deep learning is essential for safety: While computationally more expensive, deep learning models are necessary to preserve morphological fidelity and ensure clinical safety during critical transient events.
- Real-world missingness matters: Using missingness patterns derived from actual clinical data leads to more robust model evaluation than simple random masking.
- Adaptive inference is the future: A hybrid approach, using simple models for stable periods and complex models for transients, offers a computationally efficient and safe solution.
Key Results
- Stationary Efficiency: Linear interpolation achieved state-of-the-art performance (RMSE, MARD, DTW) in stable, homeostatic periods across all datasets.
- Transient Fidelity: During post-prandial peaks and hypoglycemic events, deep learning models (specifically SAITS) significantly outperformed linear interpolation in morphological fidelity (DTW) and clinical safety (MARD). For example, in Scenario B (post-prandial peaks), SAITS achieved a DTW score of 165.60 compared to Lerp's 261.82.
- RMSE Mirage Demonstrated: In Scenario B, models with similar RMSE values showed vast differences in DTW, with linear interpolation often destroying signal shape despite low pointwise error.
- Distributional Calibration: Deep learning models showed improved, but not perfect, distributional calibration during transients, meaning their imputed values were closer to the true distribution than linear interpolation, though still not perfectly aligned.
- Ecologically Valid Missingness: Models trained with empirically derived missingness patterns showed improved robustness.
Limitations and Future Work
- Model Comparisons: The study acknowledges that comparisons could be expanded to include other classical machine learning baselines (e.g., Random Forest, k-NN) and more advanced deep learning architectures (e.g., diffusion models).
- Late-Logging Phenomenon: While the study found this had a negligible impact on performance, it remains a potential challenge in real-world data.
- Downstream Task Validation: The practical utility of the proposed method in actual diabetes control systems needs further validation.
- Adaptive Inference Refinement: Future work will focus on developing a lightweight detection framework for regime-conditional model selection and optimizing for real-time latency in closed-loop control systems.
- Generalizability: While the framework is proposed to generalize to any regulated system, further empirical validation across diverse domains would be beneficial.
Summary updated