TL;DR
Existing masked autoencoders for Electronic Health Records (EHR) treat all clinical measurements as equally predictable, wasting capacity on stable biomarkers while undersampling highly volatile, clinically critical signals. This paper introduces Coefficient of Variation Masking (CV-Masking), a pretraining strategy that adaptively increases masking probabilities for inherently volatile lab tests. Combined with a value-only masking architecture, CV-Masking accelerates model convergence by 50% and systematically improves both biomarker reconstruction and downstream clinical predictions.
What Problem Does This Paper Address?
Masked Autoencoders (MAEs) are widely used to pretrain foundation models on EHR data by hiding parts of a patient's record and tasking the model with filling in the blanks. However, standard approaches rely on uniform random masking. This ignores a fundamental clinical reality: biomarkers exhibit massive heterogeneity in their natural volatility.
For example, homeostatic mechanisms keep blood sodium levels tightly regulated within a narrow, predictable range. Conversely, lactate levels can spike dramatically during acute illness or sepsis, making them highly volatile and difficult to model. Uniform masking forces models to waste capacity predicting easily guessable, stable signals, leaving them ill-equipped to capture the complex temporal patterns of volatile biomarkers that often indicate critical clinical shifts.
Methods
The authors tackle this challenge with a two-pronged approach:
- Value-Only Masked Autoencoder (VO-MAE): Instead of masking an entire clinical event triplet—which consists of the time ($t$), medical code ($c$), and numeric value ($v$)—the architecture masks only the numeric value, keeping the timing and code visible. This directly mirrors real-world clinical workflows where a doctor orders a specific test at a specific time, but the result remains unknown until the lab processes it.
- Coefficient of Variation Masking (CV-Masking): To quantify volatility fairly across different lab tests, the authors use the Coefficient of Variation ($CV = \sigma / \mu$). Unlike raw variance, CV is scale-invariant, allowing a direct comparison between features with completely different units (e.g., glucose in mg/dL vs. hemoglobin in g/dL).
During pretraining, the model uses a threshold-based binary assignment policy. Lab tests with a CV above the 75th percentile ($CV_{75}$) are designated as highly volatile and assigned a high masking weight (0.8), while the remaining tests receive a low weight (0.2). This 4:1 ratio creates a natural curriculum, forcing the model to focus its learning capacity on clinically uncertain, highly volatile signals.
Datasets
The model was trained and evaluated using the MIMIC-IV v3.1 critical care database, encompassing over 40,000 ICU admissions. The data was structured using the standardized Medical Event Data Standard (MEDS) format. The authors focused their analysis on a panel of 100 high-frequency laboratory tests (each containing more than 1,000 measurements) spanning hematology, chemistry, and urinalysis.
Key Insights and Ideas
- Volatility Correlates with Predictability: The authors confirmed a strong negative correlation (Pearson’s $r = -0.486$) between a biomarker's CV and its baseline reconstruction difficulty. Highly volatile markers are genuinely harder for models to learn.
- Curriculum Learning via Masking: By intentionally masking harder-to-predict features more frequently, CV-Masking acts as an unsupervised curriculum, guiding the model to learn complex physiological relationships rather than trivial homeostatic baselines.
- Context over Memorization: A controlled perturbation experiment revealed that CV-masked models are far more sensitive to corrupted historical data than randomly masked models. This indicates that the strategy successfully forces the network to rely on patient-specific temporal trajectories rather than simply memorizing population-level statistics.
Key Results
- Superior Reconstruction: CV-Masking achieved systematic improvements in imputation accuracy, outperforming uniform random masking on 71% of laboratory tests and variance-based masking on 68%. The largest gains were seen in immune markers, liver function tests, and metabolic indicators.
- Enhanced Clinical Predictions: When evaluating frozen representations using linear probes, CV-Masking delivered substantial improvements across downstream tasks. For instance, In-ICU Mortality prediction saw its AUROC jump from 0.682 (Random) to 0.713 (CV-Based), alongside a notable boost in AUPRC (0.107 vs 0.083), which is crucial for identifying rare but critical clinical events.
- 50% Faster Convergence: CV-Masking reached convergence in just 33 epochs, compared to 67 epochs for random masking and 100 epochs for raw variance-based masking.
Limitations and Future Work
While promising, the authors note several areas for development:
- Data Modality Limits: The evaluation was restricted to laboratory data within a single database (MIMIC-IV). Future work must validate this approach across multiple institutions and incorporate other highly volatile modalities like continuous vital signs and medication dosages.
- Static Curricula: The masking policy relies on fixed thresholds (75th percentile) and static weight ratios (0.8/0.2). Transitioning to an adaptive or learnable curriculum could further optimize training efficiency.
- Ablation Clarity: The current study does not fully disentangle the individual performance contributions of the VO-MAE architecture versus the CV-Masking policy itself.
Summary updated