Modeling Irregular Sequences with Discrete Time Tokens

15 concepts11 papers1 proposal

Standard sequence models—whether Transformers, Convolutional Neural Networks (CNNs), or Recurrent Neural Networks (RNNs)—were fundamentally designed for data with uniform spacing. In natural language, words follow one another in a strict ordinal sequence; in audio processing, signals are sampled at fixed, predictable intervals.

However, real-world event logs, particularly Electronic Health Records (EHR), are highly irregular. A patient might experience a rapid cluster of medical events during a single week of hospitalization, followed by a gap of three years before their next routine check-up.

Ignoring these gaps treats the patient journey as a simple ordered list, stripping away vital clinical context. A cough followed two days later by severe shortness of breath indicates an acute condition; the same two events separated by two years suggest entirely unrelated incidents. To capture this, models must understand the rhythm and urgency of time. Early sequence modeling approaches often struggled to integrate this non-uniform temporal spacing without resorting to complex, continuous-time mathematical frameworks that are difficult to scale.

The Core Idea: Tokenizing the Gaps

Instead of modifying the underlying neural network architecture to handle continuous time, a powerful and intuitive alternative has emerged: discretize the time intervals and treat them as tokens within the sequence itself.

By converting continuous time gaps into discrete "artificial time tokens" (such as [1_DAY], [3_MONTHS], or [LONG_TERM]), we can flatten a multi-dimensional, irregularly spaced timeline into a single, linear "sentence" of events and durations. This allows standard sequence models to process temporal dynamics natively through their existing vocabulary mechanisms.

Traditional Sequence: [Chest Pain] -> [Nitroglycerin] -> [Lab Test]
(Lacks context: Did these happen over hours or years?)

Time-Tokenized Sequence: [Chest Pain] -> [1_HOUR] -> [Nitroglycerin] -> [6_MONTHS] -> [Lab Test]

Why It Works: Intuition and Mechanism

This simple trick works because it leverages the primary strength of modern sequence models: learning contextual dependencies between tokens.

When a model like a Transformer computes self-attention over a patient's history, it learns how specific tokens modify the meaning of surrounding tokens. By inserting a time token between two medical codes, the time token acts as a contextual bridge.

For instance, in the sequence [Chest Pain] -> [1_HOUR] -> [Nitroglycerin], the model's attention mechanism learns a strong association between the symptom and the immediate intervention. If the sequence were instead [Chest Pain] -> [2_YEARS] -> [Nitroglycerin], the presence of the [2_YEARS] token naturally changes the contextual representation, signaling to the model that the medication was likely prescribed for a completely different clinical episode.

By mapping time to the token space, we gain several distinct advantages:

  1. Architectural Simplicity: We do not need to invent specialized time-aware attention layers or continuous-time mathematical frameworks. Vanilla architectures work out of the box.
  2. Non-linear Time Dynamics: Time does not always affect risk linearly. The difference between 1 hour and 2 hours can be life-threatening in an ICU, while the difference between 12 months and 13 months is negligible. Discretizing time into buckets allows the model to easily learn these non-linear thresholds.
  3. Unified Representation: The model learns joint embeddings where medical concepts and temporal intervals reside in the same latent space, capturing how the passage of time inherently interacts with disease progression.

Evolution Across Architectures

The strategy of mapping time to discrete tokens or explicit interval features has proven robust across several generations of neural network architectures.

Early Sequence Models and Embeddings

In Deepr: A Convolutional Net for Medical Records, Phuoc Nguyen et al. demonstrated that discretizing time gaps into specific categories and embedding them directly into the event sequence allowed standard CNNs to capture predictive clinical motifs without specialized architectures. Similarly, Bum Chul Kwon et al. introduced RetainEX in RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records, explicitly incorporating time intervals as input features in RNNs to distinguish high-frequency event clusters from sparse, long-term trends.

Rather than just treating time as an isolated variable, researchers found that constraining embeddings based on temporal proximity yielded richer representations. Yang Xiang et al. in Time-sensitive clinical concept embeddings learned from large electronic health records showed that grouping events into explicit temporal windows ensures that learned medical concept embeddings reflect true clinical proximity.

Attention-Based Interval Encodings

Building on this, Xueping Peng et al. developed TeSAN in Temporal Self-Attention Network for Medical Concept Embedding, demonstrating that explicitly modeling the time elapsed between visits prevents models from overlooking critical gaps in disease progression. The authors extended this concept to multi-level self-attention networks with MusaNet (Self-Attention Enhanced Patient Journey Understanding in Healthcare System), proving that explicit interval encoding is essential for mapping long-term patient journeys. In parallel, Sundreen Asad Kamal et al. developed PAVE (An interpretable risk prediction model for healthcare with pattern attention), which explicitly encodes intervals to distinguish rapid clinical declines from slow, stable symptom progressions.

Scaling to Clinical Foundation Models

With the advent of large language models and transformers, discrete temporal tokenization has become a standard paradigm for building massive healthcare foundation models.

In CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks, Chao Pang et al. formalized this by introducing specific artificial time tokens—such as 'Wn' for weeks, 'Mn' for months, and 'LT' for long-term periods—directly into a BERT-based architecture. This allowed the model to learn temporal structure without altering the core self-attention mechanism.

This tokenization strategy has enabled zero-shot capabilities and massive scaling across various modern architectures:

By treating time as a vocabulary problem rather than an architectural problem, discrete time-interval tokenization provides an elegant, scalable, and highly effective bridge between standard sequence models and the irregular realities of real-world data.

Key Foundational Work

Deepr: A Convolutional Net for Medical Records (2016) — contributed 2 concept elements to this direction.
RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_interval_encoding_time_aware_se_9942ef15,
  title = {temporal interval encoding (time-aware sequence modeling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interval-encoding-time-aware-sequence-modeling/rdr_9942ef1543d7491086abae15ca13b89f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_interval_encoding_time_aware_se_9942ef15_v1,
  title = {temporal interval encoding (time-aware sequence modeling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-interval-encoding-time-aware-sequence-modeling/rdr_9942ef1543d7491086abae15ca13b89f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake