The Tyranny of the Uniform Grid
Standard deep learning models—such as recurrent neural networks, convolutional networks, and standard Transformers—share a foundational assumption: data arrives on a uniform grid. They assume that the time elapsed ($\Delta t$) between step $t$ and step $t+1$ is constant.
In the real world, data is rarely so cooperative. Consider electronic health records (EHRs). As observed by David J. Albers et al. in Population Physiology: Leveraging Electronic Health Record Data to Understand Human Endocrine Dynamics, medical measurements are taken based on clinical necessity rather than a fixed experimental schedule. A patient in the intensive care unit (ICU) might have their heart rate recorded every few minutes during a crisis, but only once every few hours when stable. Similarly, in industrial sensor networks, hardware failures or transmission delays introduce unpredictable gaps, as noted by Md Rasel Sarkar et al. in Enhancing Wind Power Forecast Precision via Multi-head Attention Transformer: An Investigation on Single-step and Multi-step Forecasting.
When faced with such irregular temporal sampling, traditional workflows rely on preprocessing tricks like forward-filling (carrying the last observed value forward) or mean imputation. However, as Zhengling He et al. point out in Early Sepsis Prediction Using Ensemble Learning with Features Extracted from LSTM Recurrent Neural Network, simple forward-filling oversimplifies complex physiological trends and can introduce severe biases. Crucially, these methods erase the informative missingness of the data: the very fact that a doctor did not order a test is a signal that the patient is likely stable.
Furthermore, multivariate time series introduce "channel-wise asynchrony." As described by Shuhan Zhong et al. in MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification, different sensors (such as blood pressure monitors and laboratory blood panels) record data at completely different, misaligned timestamps, disrupting the continuous temporal patterns that standard architectures rely on.
To solve these challenges, researchers have developed architectures that treat time as a continuous variable rather than a discrete index.
Explicit Time Gaps and Memory Decay
The first intuitive step away from uniform grids is to make the model explicitly aware of the time elapsed between observations. If the network knows exactly how much time has passed since the last data point, it can adjust its internal state accordingly.
Instead of treating a recurrent network's hidden state as a static vault that only updates when a new token arrives, we can introduce a continuous decay mechanism. For example, in Temporal Belief Memory: Imputing Missing Data during RNN Training, Yeo Jin Kim et al. introduced a method that decays the "belief" or reliability of a historical observation based on $\Delta t$. If a patient's blood glucose was measured ten minutes ago, that measurement remains highly informative for the current prediction; if it was measured ten hours ago, its influence decays toward a baseline population mean. By feeding $\Delta t$ directly into the network architecture, the model learns to dynamically weigh past observations based on temporal distance.
Shifting to Continuous-Time Differential Equations
While decaying hidden states mitigate the problem, they still treat the underlying system as a sequence of discrete events. A more profound paradigm shift is to model the latent state of the system as a continuous flow using differential equations.
Instead of updating a hidden state via discrete transitions ($h_{t} = f(h_{t-1}, x_t)$), continuous-time models define the derivative of the hidden state with respect to time using a neural network:
$$\frac{dh(t)}{dt} = f(h(t), t)$$
This formulation forms the backbone of Neural Ordinary Differential Equations (Neural ODEs) and Neural Controlled Differential Equations (NCDEs). As demonstrated by Hyowon Wi et al. in Continuous-time Autoencoders for Regular and Irregular Time Series Imputation, a continuous-time autoencoder can encode irregular and incomplete data into a continuous "hidden path."
Why does this work so well for irregular data? Because a differential equation can be integrated over any arbitrary time interval. When an observation arrives at an irregular timestamp $t_k$, the model simply integrates the hidden path up to $t_k$ and evaluates the state. There is no need for artificial imputation or binning; the model naturally accommodates arbitrary gaps because the latent trajectory exists continuously across time. This mathematical framework is leveraged by Taylor W. Killian et al. in Continuous Time Evidential Distributions for Irregular Time Series and Yibo Yang et al. in Bayesian differential programming for robust systems identification under uncertainty to provide robust predictions and well-calibrated uncertainty estimates from highly sporadic observations.
Attention and Token Mixing Across Asynchronous Channels
Transformers offer an alternative to recurrence, but standard self-attention struggles when forced to process massive, sparsely filled timelines. To adapt attention to irregular sequences, models must decouple the sequence from rigid position embeddings and instead treat observations as a set of spatio-temporal coordinates.
In Attend and Diagnose: Clinical Time Series Analysis Using Attention Models, Huan Song et al. dispensed with recurrence entirely, utilizing a dense interpolation strategy combined with self-attention to capture multi-scale clinical patterns. To resolve the issue of channel-wise asynchrony, the multi-scale token mixing transformer (MTM) proposed by Shuhan Zhong et al. in MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification explicitly mixes tokens across different scales and channels, allowing the model to correlate sensors that do not record data at the same time.
Furthermore, when data is sparse, standard attention mechanisms waste computation on uninformative or empty intervals. To combat this, Lars Schmidt-Thieme et al. in Sparse self-attention guided generative adversarial networks for time-series generation utilized sparse self-attention transformations (like the 1.5-entmax transformation). This allows the network to focus its attention weights exclusively on the most informative historical points, effectively ignoring the vast temporal gaps that typically degrade standard attention models.
The Functional View: Gaussian Processes
Another powerful way to build intuition for irregular time series is to view the data not as a sequence of points, but as a collection of noisy samples drawn from an underlying continuous function. This is the domain of Gaussian Processes (GPs).
A Gaussian Process defines a distribution over functions, parameterized by a mean and a kernel (covariance) function. When we observe sparse, irregular data points, the GP posterior provides a continuous curve that fits the observed points while explicitly quantifying the uncertainty in the unobserved intervals between them.
In A Multivariate Timeseries Modeling Approach to Severity of Illness Assessment and Forecasting in ICU with Sparse, Heterogeneous Clinical Data, Marzyeh Ghassemi et al. utilized Multi-Task Gaussian Processes to handle messy ICU records. By fitting a GP to the sparse data, they treated the learned model hyperparameters as a compact, continuous "latent fingerprint" of the patient's physiological state. To handle extreme sparsity, Benjamin M. Marlin et al. developed a framework in Classification of Sparse and Irregularly Sampled Time Series with Mixtures of Expected Gaussian Kernels and Random Features that computes kernels over the GP posteriors themselves. This ensures that the model explicitly accounts for the high uncertainty during long unobserved gaps, preventing the classifier from making overconfident predictions when data is scarce.
Similarly, when modeling chronic conditions, Jonathan Huang in Probabilistic Event Cascades for Alzheimer's disease treats sparse, noisy snapshots of clinical decline as part of a continuous probabilistic cascade, reconstructing the precise sequence of pathological changes from sporadic measurements.
By moving away from the rigid constraints of the uniform grid—whether through memory decay, continuous differential equations, sparse attention, or Gaussian processes—modern architectures transform temporal sparsity from a missing-data problem into a continuous-time modeling opportunity.