Standard sequence models—whether Transformers, Convolutional Neural Networks (CNNs), or Recurrent Neural Networks (RNNs)—were fundamentally designed for data with uniform spacing. In natural language, words follow one another in a strict ordinal sequence; in audio processing, signals are sampled at fixed, predictable intervals.
However, real-world event logs, particularly Electronic Health Records (EHR), are highly irregular. A patient might experience a rapid cluster of medical events during a single week of hospitalization, followed by a gap of three years before their next routine check-up.
Ignoring these gaps treats the patient journey as a simple ordered list, stripping away vital clinical context. A cough followed two days later by severe shortness of breath indicates an acute condition; the same two events separated by two years suggest entirely unrelated incidents. To capture this, models must understand the rhythm and urgency of time. Early sequence modeling approaches often struggled to integrate this non-uniform temporal spacing without resorting to complex, continuous-time mathematical frameworks that are difficult to scale.
The Core Idea: Tokenizing the Gaps
Instead of modifying the underlying neural network architecture to handle continuous time, a powerful and intuitive alternative has emerged: discretize the time intervals and treat them as tokens within the sequence itself.
By converting continuous time gaps into discrete "artificial time tokens" (such as [1_DAY], [3_MONTHS], or [LONG_TERM]), we can flatten a multi-dimensional, irregularly spaced timeline into a single, linear "sentence" of events and durations. This allows standard sequence models to process temporal dynamics natively through their existing vocabulary mechanisms.
Traditional Sequence: [Chest Pain] -> [Nitroglycerin] -> [Lab Test]
(Lacks context: Did these happen over hours or years?)
Time-Tokenized Sequence: [Chest Pain] -> [1_HOUR] -> [Nitroglycerin] -> [6_MONTHS] -> [Lab Test]
Why It Works: Intuition and Mechanism
This simple trick works because it leverages the primary strength of modern sequence models: learning contextual dependencies between tokens.
When a model like a Transformer computes self-attention over a patient's history, it learns how specific tokens modify the meaning of surrounding tokens. By inserting a time token between two medical codes, the time token acts as a contextual bridge.
For instance, in the sequence [Chest Pain] -> [1_HOUR] -> [Nitroglycerin], the model's attention mechanism learns a strong association between the symptom and the immediate intervention. If the sequence were instead [Chest Pain] -> [2_YEARS] -> [Nitroglycerin], the presence of the [2_YEARS] token naturally changes the contextual representation, signaling to the model that the medication was likely prescribed for a completely different clinical episode.
By mapping time to the token space, we gain several distinct advantages:
- Architectural Simplicity: We do not need to invent specialized time-aware attention layers or continuous-time mathematical frameworks. Vanilla architectures work out of the box.
- Non-linear Time Dynamics: Time does not always affect risk linearly. The difference between 1 hour and 2 hours can be life-threatening in an ICU, while the difference between 12 months and 13 months is negligible. Discretizing time into buckets allows the model to easily learn these non-linear thresholds.
- Unified Representation: The model learns joint embeddings where medical concepts and temporal intervals reside in the same latent space, capturing how the passage of time inherently interacts with disease progression.
Evolution Across Architectures
The strategy of mapping time to discrete tokens or explicit interval features has proven robust across several generations of neural network architectures.
Early Sequence Models and Embeddings
In Deepr: A Convolutional Net for Medical Records, Phuoc Nguyen et al. demonstrated that discretizing time gaps into specific categories and embedding them directly into the event sequence allowed standard CNNs to capture predictive clinical motifs without specialized architectures. Similarly, Bum Chul Kwon et al. introduced RetainEX in RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records, explicitly incorporating time intervals as input features in RNNs to distinguish high-frequency event clusters from sparse, long-term trends.
Rather than just treating time as an isolated variable, researchers found that constraining embeddings based on temporal proximity yielded richer representations. Yang Xiang et al. in Time-sensitive clinical concept embeddings learned from large electronic health records showed that grouping events into explicit temporal windows ensures that learned medical concept embeddings reflect true clinical proximity.
Attention-Based Interval Encodings
Building on this, Xueping Peng et al. developed TeSAN in Temporal Self-Attention Network for Medical Concept Embedding, demonstrating that explicitly modeling the time elapsed between visits prevents models from overlooking critical gaps in disease progression. The authors extended this concept to multi-level self-attention networks with MusaNet (Self-Attention Enhanced Patient Journey Understanding in Healthcare System), proving that explicit interval encoding is essential for mapping long-term patient journeys. In parallel, Sundreen Asad Kamal et al. developed PAVE (An interpretable risk prediction model for healthcare with pattern attention), which explicitly encodes intervals to distinguish rapid clinical declines from slow, stable symptom progressions.
Scaling to Clinical Foundation Models
With the advent of large language models and transformers, discrete temporal tokenization has become a standard paradigm for building massive healthcare foundation models.
In CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks, Chao Pang et al. formalized this by introducing specific artificial time tokens—such as 'Wn' for weeks, 'Mn' for months, and 'LT' for long-term periods—directly into a BERT-based architecture. This allowed the model to learn temporal structure without altering the core self-attention mechanism.
This tokenization strategy has enabled zero-shot capabilities and massive scaling across various modern architectures:
- Zero-Shot Simulation: Pawel Renc et al. developed ETHOS (Zero shot health trajectory prediction using transformer), inserting dedicated duration tokens to simulate health outcomes and predict diverse risks without task-specific fine-tuning.
- Early Intervention: Hyunwoo Sohn et al. utilized explicit time-gap tokens in Multi-TA (Multi-TA: Multilevel Temporal Augmentation for Robust Septic Shock Early Prediction) to capture the clinical significance of timing in rapid-onset conditions like septic shock, proving that the timing of interventions is as significant as the interventions themselves.
- Long-Context Robustness: Michael Wornow et al. leveraged artificial time tokens within subquadratic long-context models like Mamba (Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data), demonstrating that explicit interval tokenization makes models robust against the messy, irregular nature of long-term longitudinal data up to 16,000 tokens.
- Large-Scale Scaling Laws: Liwen Sun et al. scaled this approach to a 2.4-billion parameter transformer framework called CATCH-FM (Intercept Cancer: Cancer Pre-Screening with Large Scale Healthcare Foundation Models), combining discretized time tokens with positional embeddings to map out the multi-year longitudinal dynamics required for early cancer detection.
By treating time as a vocabulary problem rather than an architectural problem, discrete time-interval tokenization provides an elegant, scalable, and highly effective bridge between standard sequence models and the irregular realities of real-world data.
Key Foundational Work
Deepr: A Convolutional Net for Medical Records (2016) — contributed 2 concept elements to this direction.
RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records (2018) — contributed 1 concept elements to this direction.