Continuous Temporal Encodings for Irregularly Sampled Sequences

20 concepts19 papers1 proposal

Standard sequence models, such as Transformers and Recurrent Neural Networks (RNNs), operate under an implicit assumption: data arrives on a uniform, discrete grid. In natural language processing, word $n+1$ immediately follows word $n$. However, real-world data rarely conforms to a neat, synchronized clock.

Consider a patient's Electronic Health Record (EHR). A patient might visit a clinic twice in one week due to an acute infection, remain healthy for six months, and then undergo a routine checkup. If we feed this sequence into a standard Transformer using traditional ordinal positional encodings ($1, 2, 3$), the model treats the six-month gap identically to the one-week gap. This structural blindness discards critical context: the density and spacing of events often carry as much predictive power as the events themselves.

To bridge this gap, a major research direction has emerged that replaces discrete index-based positioning with continuous temporal encodings. By mapping continuous timestamps and intervals directly into high-dimensional vector spaces, these techniques allow deep learning architectures to natively interpret the irregular rhythms of longitudinal data.


The Core Intuition: Mapping Time to Geometry

The fundamental flaw of discrete positional encoding in irregular sequences is that it preserves order but destroys topology. If a model only knows that Event B happened after Event A, it cannot distinguish between a rapid burst of activity and a sparse set of interactions spread across years.

Continuous temporal encoding solves this by treating time as a continuous coordinate. Instead of assigning an integer index to an observation, the architecture calculates a vector representation based on the actual elapsed time ($\Delta t$) between events or their absolute position on a calendar ($t$).

When these continuous time vectors are added to or combined with feature embeddings, the geometric distance between vectors in the latent space reflects the actual physical time that passed between the real-world events. This enables the model's attention mechanism to naturally scale its weights based on temporal proximity, learning whether an event from months ago is still relevant to the present moment.


Mechanism 1: Projecting Continuous Time via Sinusoids and MLPs

To transform a raw scalar timestamp or interval into a high-dimensional vector that a neural network can utilize, researchers rely on two primary mathematical mechanisms: periodic functions and learnable projections.

Sinusoidal Encodings

Extending the classic positional encodings introduced in early Transformer architectures, several works swap out the discrete sequence index for a continuous time scalar within a set of sine and cosine functions. For instance, Patra et al. utilize continuous sinusoidal encodings to capture irregular sampling intervals in industrial thermal imaging sequences for solar power plants. Similarly, Wald et al. employ periodic functions to encode both absolute timestamps and relative durations between consecutive events, allowing sequence models to learn complex, time-dependent patterns without forcing the data onto a discrete grid.

Learnable Multi-Layer Perceptrons (MLPs)

Rather than relying on fixed periodic formulas, an alternative approach passes raw time intervals through a learnable neural network layer to project them into the embedding space. Labach et al. demonstrate this in their dual-attention architecture for EHR data, mapping continuous time intervals into high-dimensional embeddings to preserve precise event durations. In the domain of audio-visual action recognition, Chalk et al. use an MLP to map arbitrary start and end time pairs into continuous temporal interval encodings, treating event durations as first-class citizens within the model.


Mechanism 2: Balancing Absolute Dates and Relative Intervals

Time-series data often exhibits two distinct types of temporal context: absolute context (where an event sits on a macro calendar) and relative context (how much time has elapsed since the last observation). Successful architectures frequently leverage both.

Absolute and Calendar-Aware Indexing

For systems influenced by seasonal or environmental cycles, knowing the exact calendar date is paramount. In satellite image time-series classification, cloud cover and orbital paths create highly irregular observation intervals. To counteract this, Sainte Fare Garnot et al. and Tarasiou et al. map observations directly to specific calendar days of the year rather than sequential indices. This date-aware positioning ensures that the model correctly interprets crop growth stages relative to seasonal patterns, regardless of missing or unevenly spaced data points.

Relative Interval Encodings

In behavioral and clinical sequences, the relative gap between events often dictates the meaning of the data. Zhang et al. show that substituting standard relative positional encodings with absolute date and interval embeddings allows user-behavior models to distinguish between rapid bursts of mobile app usage and sparse interactions over long periods.

In medical forecasting, this interval representation is critical for tracking disease progression. Long et al. explicitly incorporate learned embeddings of elapsed time between medical events to capture clinical context. This strategy is mirrored by Zhong et al. to capture longitudinal health risks in diffusion models, and by Wang et al. to track disease progression across a patient's historical radiology reports. Additionally, Holste et al. calculate temporal positional encodings based on the actual time elapsed since a patient's baseline medical image, significantly improving long-term eye disease prognosis.


Why It Works: Bypassing the Imputation Bottleneck

Historically, the standard recipe for handling irregularly sampled time series involved a preprocessing step: either resampling the data onto a uniform fixed-grid or using statistical imputation to fill in missing values. Both methods are problematic. Fixed-grid resampling introduces discretization errors, while imputation can invent data that distorts the true underlying signal.

Continuous temporal encodings work because they bypass the imputation bottleneck entirely. By treating time as a continuous feature, models can ingest raw, asynchronous data directly.

Traditional Fixed-Grid Imputation:
Raw Data:   [Event A]-------(3 months)-------[Event B]--(1 week)--[Event C]
Imputed:    [Event A][  NaN  ][  NaN  ][  NaN  ][Event B][Event C][  NaN  ]  <-- Wasted compute & artifacts

Continuous Temporal Encoding:
Raw Data:   [Event A]                        [Event B]            [Event C]
Time Enc:   [ t = 0 ]                        [ t = 90]            [ t = 97]  <-- Exact geometry preserved

This paradigm shift manifests in several key architectural advantages:

  • Direct Asynchronous Processing: Li et al. demonstrate that learnable time embeddings map irregular timestamps into a continuous vector space, eliminating the need for fixed-grid imputation in messy clinical data. Taking this further, Patel et al. format irregular sequences as triplets consisting of a timestamp, a feature index, and a value, allowing transformers to process asynchronous data natively.
  • Time-Aware Attention Lifespans: When continuous time embeddings interact with attention mechanisms, the model can learn mathematical "decay" or "relevance windows." As Cai et al. demonstrate, incorporating the actual elapsed time allows attention mechanisms to discover that different medical conditions have distinct temporal lifespans, adjusting their predictive influence accordingly.
  • Unlocking Pre-trained Models: This geometric representation of time is robust enough to adapt architectures designed for other modalities. Zhang et al. successfully adapted large pre-trained language models (like GPT-2) to irregular time-series tasks by replacing discrete positional markers with continuous-time embeddings, enabling precise mathematical interpolation and extrapolation without altering the underlying language model weights.
  • Dynamic Contextual Prompts: In streaming environments where data distributions shift over time, continuous encodings help maintain historical context. Xue et al. utilize continuous-time positional encodings calculated from average event durations to dynamically adjust retrieval prompts, allowing neural temporal point processes to adapt to streaming sequences without catastrophic forgetting.

By treating time as a continuous, geometric dimension rather than a discrete sequence index, these architectures ensure that deep learning models remain accurate, efficient, and faithful to the true temporal dynamics of real-world data.

Key Foundational Work

General-Purpose User Embeddings based on Mobile App Usage (2020) — contributed 2 concept elements to this direction.
BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes (2020) — contributed 1 concept elements to this direction.
Medical Concept Embedding with Time-Aware Attention (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_interval_encoding_irregular_tem_11ce486a,
  title = {temporal interval encoding (irregular temporal interval encoding)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interval-encoding-irregular-temporal-interval-encoding/rdr_11ce486a2fce47619efd55e10a259421},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_interval_encoding_irregular_tem_11ce486a_v1,
  title = {temporal interval encoding (irregular temporal interval encoding)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interval-encoding-irregular-temporal-interval-encoding/rdr_11ce486a2fce47619efd55e10a259421/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake