Continuous Functional Time Encodings for Neural Networks

20 concepts18 papers1 proposal

In standard sequential deep learning, time is often treated as a discrete sequence of indices. When processing text, the first word is at position 1, the second at position 2, and so on. This ordinal approach underpins the classic positional encodings used in architectures like the Transformer.

However, real-world data rarely conforms to a neat, evenly spaced grid. In medical health records, climate science, and financial logs, observations occur at irregular intervals (Luo et al.; Shukla and Marlin). In social networks and knowledge graphs, interactions happen asynchronously in continuous time (Zuo et al.; Kazemi et al.).

If we force these continuous timestamps into discrete bins or simple ordinal indices, we discard vital information: the exact duration between events. Conversely, feeding raw scalar timestamps directly into a neural network fails because deep architectures struggle to learn complex, non-linear, or periodic temporal patterns from a single raw number.

To bridge this gap, a powerful paradigm has emerged: functional time encoding. Instead of treating time as a discrete index or a raw scalar, this approach maps continuous timestamps or intervals into high-dimensional vector spaces using a mathematically grounded basis of continuous functions.


The Geometric Goal: Translation Invariance

To understand why functional time encoding works, we must first define what we want from a temporal embedding space. If two events occur with a time gap of $\Delta t$, the geometric relationship between their embeddings should reflect that specific interval, regardless of when the sequence began.

In machine learning terms, we want a coordinate mapping function $\Phi(t)$ that yields a translation-invariant (or shift-invariant) kernel. As established by Xu et al. (neurips 2019), the inner product of two temporal embeddings should depend solely on the displacement between them:

$$\langle \Phi(t), \Phi(s) \rangle = \mathcal{K}(t - s)$$

By satisfying this property, the geometric distance between vectors in the embedding space accurately preserves the actual physical time spans between events. This allows downstream neural network operations—such as the dot-product attention mechanism in Transformers—to naturally compute dependencies based on exact temporal intervals rather than arbitrary sequence positions.


The Mathematical Engine: Bochner's Theorem

How do we construct a practical mapping function $\Phi(t)$ that guarantees this translation-invariant property? The answer lies in Bochner’s Theorem, a classical result from harmonic analysis.

Bochner’s Theorem states that any continuous, positive-definite, shift-invariant kernel $\mathcal{K}(t-s)$ can be represented as the Fourier transform of a positive finite measures. In practice, this means we can approximate any such temporal kernel by projecting the scalar time value onto a truncated basis of periodic sine and cosine functions (Xu et al. (neurips 2019); Xu et al. (iclr 2020)).

A standard functional time encoding vector takes the following form:

$$\Phi(t) = \sqrt{\frac{2}{d}} \left[ \cos(\omega_1 t), \sin(\omega_1 t), \dots, \cos(\omega_k t), \sin(\omega_k t) \right]^T$$

Here, $\omega_1, \dots, \omega_k$ represent different frequencies. This formulation provides several immediate advantages:

  1. Continuous Interpolation: Because sine and cosine are continuous functions, $\Phi(t)$ can evaluate any arbitrary, real-valued timestamp, allowing models to seamlessly handle irregular intervals (Kazemi et al.).
  2. Extrapolation: The model can naturally generalize to unseen future timestamps during inference, which is crucial for forecasting tasks (Sun et al.; Liu).
  3. Expressiveness: By combining multiple frequencies, the network can capture both high-frequency localized dynamics and low-frequency long-term trends or seasonal cycles (Zhang et al.).

While some implementations use fixed, pre-defined frequencies (similar to the original Transformer's positional encodings), many modern frameworks treat the frequencies $\omega_i$ as learnable parameters, optimizing them via backpropagation alongside the rest of the network (Shukla and Marlin; Sun et al.).


Integration Across Architectures

Functional time encodings are highly versatile. Because they output a standard vector of dimension $d$, they can be easily integrated into existing deep learning architectures via concatenation or addition.

1. Self-Attention and Transformers

In a standard Transformer, attention weights are calculated using queries and keys derived from token embeddings. By adding or concatenating the functional time embedding $\Phi(t)$ to the token representation, the attention matrix becomes explicitly time-aware.

For instance, Zuo et al. utilize this to model asynchronous event sequences via a Transformer Hawkes Process, allowing the network to calculate how the intensity of future events decays or amplifies over continuous intervals. Similarly, Shukla and Marlin leverage continuous time embeddings within Multi-Time Attention Networks (mTANs) to efficiently query irregular time series at arbitrary target time points.

2. Dynamic Graph Neural Networks

Graphs that evolve over time present a unique challenge: edges and nodes appear and disappear asynchronously. Rather than snapshotting the graph into discrete time steps, continuous-time dynamic graph models use functional encodings to tag each interaction with its exact timestamp (Yang; Yi et al.).

In the Temporal Graph Attention (TGAT) network proposed by Xu et al. (iclr 2020), functional time encodings are integrated directly into the node aggregation step. This allows the graph attention mechanism to weigh a neighbor's influence based not just on topology, but on how recently the interaction occurred.

Raw Timestamp (t) ───► [ Bochner's Theorem Expansion ] ───► Continuous Vector Φ(t)
                                                                   │
                                                                   ▼
Node/Item Features ────────────────────────────────────────► [ Concatenate / Add ]
                                                                   │
                                                                   ▼
                                                       Time-Aware Neural Layer
                                                    (Attention / Graph Message)

3. Industrial Recommendation Systems

In sequential recommendation, the gap between user actions provides critical context. A user buying a coffee 5 minutes after looking at a pastry behaves differently than a user looking at a pastry 5 weeks later.

To capture this, industrial recommendation architectures utilize translation-invariant functional encodings to map the time gaps between user clicks (Xie et al. (sigir 2022); Sguerra et al.). By embedding these continuous intervals, models like the one developed by Xie et al. (arxiv 2023) can preserve evolving temporal patterns and user preferences without relying on coarse, hand-crafted time buckets.


Why It Works: A Summary of Intuition

Functional time encoding succeeds because it shifts the burden of temporal reasoning away from the network's deep layers and handles it directly within the representation space.

By mapping scalar time to a vector space via a Fourier-like functional basis, it satisfies the mathematical requirement of shift-invariance. The network no longer has to learn how to subtract timestamps or deduce periodicity from scratch; instead, the geometric relationships—the angles and distances between vectors—already perfectly mirror the physical reality of continuous time.

Key Foundational Work

Self-attention with Functional Time Representation Learning (2019) — contributed 3 concept elements to this direction.
Inductive representation learning on temporal graphs (2020) — contributed 1 concept elements to this direction.
Transformer Hawkes Process (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_functional_time_encoding_3f8b4d4c,
  title = {functional time encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-time-encoding/rdr_3f8b4d4c913443f3b2c5209e2b524062},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_functional_time_encoding_3f8b4d4c_v1,
  title = {functional time encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-time-encoding/rdr_3f8b4d4c913443f3b2c5209e2b524062/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake