Large Language Models (LLMs) possess remarkable sequential reasoning capabilities, but they are fundamentally optimized for discrete, symbolic tokens organized by syntax and grammar. Conversely, real-world physical, medical, and financial phenomena manifest as continuous, high-frequency numerical time series defined by mathematical properties like trends, seasonality, and sudden anomalies.
This stark contrast creates a profound modality gap (Large Language Models for Time Series: A Survey). When we try to force an LLM to read raw numbers, it treats them as isolated text strings, losing the underlying geometric and physical dynamics (CC-Time: Cross-Model and Cross-Modality Time Series Forecasting). Bridging this representation gap requires mechanisms that translate continuous numerical sequences into spaces where language models can natively reason.
Why Naive Tokenization Fails
If you prompt an LLM with a string of numbers like 1.23, 1.45, 1.67, a standard text tokenizer breaks these characters down into arbitrary sub-word tokens (e.g., "1", ".2", "3"). The model has no inherent understanding that 1.23 is quantitatively close to 1.24; to its embedding space, these tokens are as distinct as "apple" and "banana".
Furthermore, representing continuous values as text strings causes severe computational inefficiency and a high rate of temporal hallucinations (Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models). As highlighted by Zhe Li et al. in FoundTS: Comprehensive and Unified Benchmarking of Foundation Models for Time Series Forecasting, this lack of alignment causes interference between pre-trained textual knowledge and numerical requirements, sometimes yielding performance worse than random initialization. Individual scalar values lack inherent semantic meaning on their own (Skip-Timeformer: Skip-Time Interaction Transformer for Long Sequence Time-Series Forecasting); they only gain meaning through their broader geometric context.
Reprogramming the Token Space
Instead of converting numbers to text strings, a more effective approach maps continuous time-series patches directly into the LLM's native embedding space.
Continuous Time Series ──> [ Patching ] ──> [ Linear Projector ] ──> Reprogrammed Tokens (LLM Space)
Ming Jin et al. pioneered this in Time-LLM: Time Series Forecasting by Reprogramming Large Language Models by grouping continuous time steps into patches and using a linear layer to "reprogram" them into representations that mimic text embeddings. This allows the frozen language model to process the signal as if it were a sequence of words.
Taking this a step further, Filippos Bellos et al. introduced vocabulary inversion in VITRO: Vocabulary Inversion for Time-series Representation Optimization. Instead of relying on standard text tokens, the authors learn dataset-specific "pseudo-words" that capture seasonal patterns and cyclical trends. By expanding the LLM's vocabulary with specialized numerical tokens (ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data), these methods allow models to natively interpret quantitative relationships without destroying the underlying signal structure.
Semantic Anchoring and Distribution Alignment
Reprogramming the input layer is only half the battle; the model's internal representations must also align with the semantic concepts it already understands.
Zijie Pan et al. address this in $S^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting by mapping temporal patterns to "semantic anchors" derived from pre-trained word embeddings. For example, a sudden upward spike in a time series is explicitly mapped near the embedding space of words like "surge" or "growth". This grounds raw quantitative sequences in the LLM's pre-trained linguistic knowledge base.
To ensure this alignment holds true deeper within the network, Hang Guo et al. developed CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning. This framework explicitly aligns the statistical distributions of time series data with textual representations at three distinct levels:
- Input level: Mapping patches to token spaces.
- Intermediate level: Ensuring attention maps align with temporal dependencies.
- Output level: Aligning the final forecasting representations.
This multi-level alignment prevents the model from drifting into superficial statistical correlations, forcing it to apply its cross-modal reasoning capabilities to the forecasting task.
Continuous Vector In-Context Learning
Can we bypass textual tokenization entirely? Yufan Zhuang et al. demonstrate in Vector-ICL: In-context Learning with Continuous Vector Representations that lightweight projectors can map continuous vector representations directly into an LLM's embedding space.
This approach reveals that the core architecture of a Transformer—its attention mechanism—is fundamentally a powerful pattern matcher that does not strictly require text. By feeding the model continuous vectors alongside a few demonstration examples, the LLM can perform in-context learning on raw signals (such as fMRI or sensor data) without any text translation layer.
Bypassing Text via Vision Models
A radical alternative to resolving the text-token dilemma is to abandon language models altogether in favor of vision models. Text-based tokenization often fails to preserve the spatial and frequency relationships inherent in signals (Cross-Modal Knowledge Transfer in Time Series AI via Large Vision Models).
By encoding 1D time series into 2D images (such as spectrograms or recurrence plots), researchers can leverage Large Vision Models (LVMs) instead (Harnessing Vision Models for Time Series Analysis: A Survey).
1D Time Series ──> [ 2D Visual Encoding ] ──> [ Pre-trained ViT ] ──> Image Inpainting (Forecast)
For instance, Mouxiang Chen et al. show in VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters that treating forecasting as a visual "image inpainting" task allows pre-trained vision models to outperform purpose-built time series foundation models. Because vision models are trained to understand continuous spatial gradients and structures, they naturally bypass the discrete tokenization bottleneck that hinders language-based architectures.
Key Foundational Work
: pitfalls of representing and tokenizing temporal data for large language models (2024) — contributed 3 concept elements to this direction.
VITRO: Vocabulary Inversion for Time-series Representation Optimization (2024) — contributed 2 concept elements to this direction.
Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Era (2024) — contributed 2 concept elements to this direction.