Abstract
Irregular time series (ITS) are a hallmark of real-world data, particularly in clinical environments where patient measurements are recorded at uneven intervals and across asynchronous variables. Traditional self-supervised learning (SSL) methods, such as random masking, often fail to capture the underlying physiological dynamics inherent in these irregularities. We present EMIT (Event-Based Masked Auto Encoding), a novel pretraining framework designed to leverage the "irregularity signal." By prioritizing the reconstruction of significant events—defined by their rate of change—EMIT learns to capture critical clinical fluctuations more effectively than standard approaches. Our results on gold-standard ICU datasets demonstrate that EMIT significantly outperforms existing baselines, especially in scenarios with limited labeled data.
Introduction
In healthcare, data collection is rarely uniform. A patient's vital signs may be monitored continuously, while laboratory results are updated sporadically. This results in irregular time series characterized by uneven gaps and missing data. Existing SSL methods often treat these series as generic sequences, applying random masking techniques borrowed from natural language processing or computer vision. However, in medicine, the timing and velocity of a measurement change are often as diagnostic as the value itself.
EMIT addresses this by treating irregularity not as noise, but as a primary signal. By focusing on "significant events"—sudden spikes or drops in physiological markers—the model is forced to understand the dynamics of a patient's health rather than just interpolating between static points.
Methodology
1. Event-Based Masking Strategy
The core innovation of EMIT is its masking logic. Instead of masking a fixed percentage of data points at random, the model calculates the rate of change (the slope) between consecutive data points for every variable.
- Significant Events: Points with high slopes (rapid changes).
- Insignificant Events: Points with low slopes (stable patterns).
EMIT employs a probabilistic approach to mask both types of events, ensuring the model learns to reconstruct critical fluctuations while maintaining a baseline understanding of normal physiological states.
2. Latent Space Reconstruction
Unlike traditional Masked Autoencoders (MAEs) that attempt to reconstruct raw input values, EMIT operates in the latent space. It masks the internal mathematical embeddings—representations of time, feature, and value—and tasks the decoder with reconstructing these hidden representations. This prevents the model from "cheating" through simple linear interpolation of raw values and encourages the development of more robust, generalized features.
3. Architecture and Training
The framework utilizes a Transformer-based encoder inspired by the STraTS architecture, which explicitly processes triplets of (time, feature, value). The model is trained using a dual loss function: a reconstruction loss (Mean Squared Error between original and predicted embeddings) and an auxiliary forecasting loss to improve temporal awareness.
Experimental Setup
Datasets
EMIT was evaluated on two prominent clinical datasets for mortality prediction:
- MIMIC-III: Records from over 40,000 ICU patients.
- PhysioNet Challenge 2012: EHR data from over 10,000 ICU stays.
To simulate the data-scarce reality of medical research, the researchers tested the model's performance using only a fraction (10% to 50%) of available labels during the fine-tuning phase.
Results and Discussion
EMIT consistently outperformed a wide range of baselines, including RNN-based models (GRU-D), Convolutional models (TCN), and other Transformer-based architectures (SeFT, STraTS).
Performance Metrics
- MIMIC-III (50% labels): EMIT achieved a PR-AUC of 0.578, a significant improvement over the previous state-of-the-art baseline, STraTS (0.558).
- PhysioNet 2012 (50% labels): EMIT reached a ROC-AUC of 0.846, compared to 0.828 for STraTS.
The Importance of Event-Based Masking
Ablation studies confirmed that the "Event-based" masking strategy consistently outperformed "Random" masking across all metrics. This suggests that what is masked is just as critical as how much is masked. By focusing on high-rate-of-change points, the model extracts more informative features from unlabeled data, making it highly efficient in label-scarce environments.
Conclusion and Future Work
EMIT demonstrates that incorporating domain-specific knowledge—such as the importance of physiological rate of change—into self-supervised pretraining can yield substantial gains for irregular time series analysis. While currently focused on healthcare, the underlying logic of event-based masking is potentially applicable to other domains like finance or IoT.
Future research will focus on scaling EMIT to handle exceptionally long time series and optimizing the computational cost, as current pretraining requires substantial GPU resources (4 to 7 hours on H100/A100 hardware). Additionally, exploring the transferability of the "rate of change" threshold across different domains remains a key area for investigation.
Summary updated