Historically, time series forecasting, generation, and classification have been treated as continuous regression problems. Models are typically trained to minimize Mean Squared Error (MSE) or Mean Absolute Error (MAE), forcing them to predict the exact future amplitude of a signal.
However, a powerful paradigm shift has emerged across the machine learning landscape. Instead of treating time series as continuous numerical streams, researchers are transforming them into sequences of discrete tokens—effectively mapping numbers to a "vocabulary" in the same way large language models (LLMs) process text. By bridging this modality gap, standard language modeling architectures can be applied directly to temporal data, unlocking remarkable zero-shot capabilities, robust denoising, and flexible probabilistic generation.
The Mechanics of Time Series Tokenization
To treat a time series like natural language, continuous values must be mapped to a finite set of discrete tokens. Across recent literature, this discretization happens via two primary methodologies: value-based uniform binning and learned vector quantization.
Value-Based Uniform Binning
The most direct way to tokenize a time series is to divide its numerical range into a fixed number of bins. In this setup, each bin corresponds to a specific token in a vocabulary.
For instance, in the Chronos framework, Ansari et al. apply a mean-scaling operation to a time series segment and then partition the scaled values into uniform historical bins. This maps each individual data point to a categorical token, allowing a standard T5 architecture to perform autoregressive next-token prediction. Similarly, Tai et al. utilize uniform binning to allow categorical language models to process non-textual continuous sequences.
To handle data with highly skewed distributions or varying scales, Régaldo-Saint Blancard et al. introduce a variation in the AION-1 architecture: they apply uniform binning to the cumulative distribution functions (CDFs) of numerical data. This ensures that the resulting tokens represent equal probability mass, allowing sequence models to seamlessly ingest scalar values across vastly different astronomical scales.
Learned Vector Quantization
Instead of tokenizing individual data points based on their raw values, a more sophisticated approach encodes entire temporal segments or patterns into a learned codebook. This is typically achieved using a Vector Quantized Variational Autoencoder (VQ-VAE).
Continuous Signal ---> [ Temporal Encoder ] ---> Continuous Embeddings
|
v
Discrete Token Sequence <--- [ Codebook Lookup ] <--- Quantization Layer
In this paradigm, an encoder compresses a window of the time series into a continuous embedding, which is then mapped to the closest vector in a learned codebook. Tao et al. leverage this in CrossTimeNet, combining VQ-VAEs with Temporal Convolutional Networks (TCNs) to compress time series into uniform representations that capture underlying temporal dynamics.
This segment-level tokenization can also be tailored to specific domains. For example, Pradeepkumar et al. introduce a framework that tokenizes single-channel EEG signals by learning a vocabulary of "time-frequency motifs." In the domain of biology, Xu et al. utilize a VQ-VAE to discretize continuous neural traces from calcium imaging into a discrete vocabulary, enabling autoregressive transformers to model population dynamics.
Why Discretization Works: Building Intuition
At first glance, throwing away numerical precision by binning or quantizing data seems counterintuitive. Why would a model perform better when forced to view a precise floating-point number as a coarse categorical token? The answer lies in how discretization changes the loss landscape, filters noise, and unlocks architectural priors.
1. Expressing Multi-Modal Probabilities
When a model is trained with a continuous regression loss like MSE, it is mathematically incentivized to predict the mean of all possible future outcomes. If a future trend is highly uncertain—for example, if a stock price or traffic pattern might sharply spike or sharply drop—an MSE-optimized model will split the difference and predict a flat, highly unrealistic average line.
By tokenizing the time series, the problem is reframed as a classification task optimized via cross-entropy loss. As highlighted by Serapio et al., this allows the model to output a full probability distribution over the vocabulary of future values. The model can express multi-modal beliefs (e.g., "there is a 40% chance of a spike and a 60% chance of a drop"), enabling much more realistic and flexible probabilistic forecasting.
2. Inherent Denoising
Real-world time series data is notoriously noisy, filled with high-frequency jitter and stochastic fluctuations that do not contribute to the underlying trend. Continuous models often waste capacity trying to fit this micro-noise.
Zhao et al. demonstrate that representing continuous data as discrete tokens serves as an inherent denoising mechanism. Because the quantization step clusters similar data segments into the same codebook vectors, it effectively filters out low-amplitude stochastic noise. The downstream generative model can then focus entirely on the macro-level grammar of the temporal transitions without getting distracted by high-frequency jitter.
3. Unlocking the Sequential Prior of Language Models
Time series data and natural language share a fundamental property: they are governed by sequential dependencies, structural repetitions, and multi-scale patterns.
As Tai et al. show, large-scale language pretraining equips models with a powerful "sequential prior." When we tokenize a time series, we bridge the modality gap (Zhang et al.; Jiang et al.), allowing us to feed temporal data directly into off-the-shelf Transformer architectures. The model can immediately repurpose its learned attention mechanisms to track long-range dependencies and multi-scale structures in the tokenized numbers.
Applications and Generalization
Once a time series is tokenized, it can be manipulated using the same generative techniques that power modern LLMs. This has led to breakthroughs in generation speed, cross-domain transfer, and zero-shot forecasting.
High-Fidelity Generation
Traditional generative models for time series, such as diffusion models, require iterative denoising steps that are computationally expensive. Token-based architectures generate data autoregressively, which can drastically accelerate synthesis. For instance, Chen et al. introduce SDformer, a similarity-driven discrete transformer that generates high-quality synthetic time series up to 30 times faster than state-of-the-art diffusion benchmarks. Similarly, Li et al. present FAR-TS, which factorizes multivariate signals into a spatial basis and quantized temporal tokens, enabling rapid, arbitrary-length sequence generation.
Zero-Shot Forecasting of Complex Systems
Because discrete tokenization unifies how a model views different data streams, models trained on massive corpora of tokenized time series display remarkable zero-shot generalization.
A striking example of this is documented by Zhang et al., who evaluated large time-series foundation models on chaotic, non-linear dynamical systems. They discovered that tokenized models could forecast chaotic systems zero-shot, matching the short-term accuracy of custom-trained benchmarks. More importantly, even after individual point predictions inevitably diverged due to the butterfly effect, the tokenized models remarkably preserved the long-term geometric structure and statistical properties (the "strange attractor") of the chaotic system.
The Precision Trade-Off
While discrete tokenization offers massive advantages in stability, speed, and multimodal modeling, it does introduce a core trade-off: quantization error. As noted by Emami et al. in their evaluation of short-term load forecasting on the BuildingsBench dataset, discretizing continuous numerical sequences into a fixed vocabulary can sometimes sacrifice fine-grained precision compared to pure continuous regression methods in high-resolution scenarios.
The field continues to evolve by finding optimal balances—such as the hierarchical approach by Zhao et al. that predicts a coarse long-term trend before generating high-resolution details—proving that treating numbers as language is a robust, scalable path forward for temporal AI.
Key Foundational Work
Tabular Transformers for Modeling Multivariate Time Series (2020) — contributed 1 concept elements to this direction.
BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecasting (2023) — contributed 1 concept elements to this direction.