Cross-Modal Alignment for Time Series Foundation Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2210 papers237 conceptsfrozen 2026-10-04

Large Language Models (LLMs) have demonstrated an uncanny ability to reason, generalize, and perform zero-shot tasks across diverse textual domains. However, a significant frontier remains: time series data. Whether it is the fluctuating pulse of an EKG, the high-frequency oscillations of an industrial sensor, or the complex trends of global weather, these signals are fundamentally different from the discrete, symbolic nature of human language.

The emerging research direction in this field focuses on cross-modal alignment—developing mechanisms to "reprogram" or "translate" continuous numerical signals into the latent semantic spaces of pre-trained foundation models. The goal is not to build new models from scratch, but to unlock the latent reasoning power of existing LLMs and Vision Transformers (ViTs) for temporal analysis.

The Modality Gap

The primary obstacle to using LLMs for time series is the "modality gap." As noted by Ming Jin and colleagues in Time-LLM, there is a fundamental misalignment between continuous, multi-dimensional floating-point values and the discrete tokens used in natural language processing.

Language models are trained to predict the next word in a sequence based on semantic relationships. Time series, by contrast, are governed by physical dynamics, seasonal patterns, and statistical noise. Without a bridge, an LLM sees a sequence of numbers as a collection of meaningless symbols rather than a coherent trend. Wei Li argues in Prioritizing Alignment Paradigms that we must prioritize these alignment paradigms over task-specific customization to truly leverage the "reasoning" capabilities of foundation models.

Reprogramming and Semantic Anchoring

To bridge this gap, researchers have developed "reprogramming" techniques. Instead of fine-tuning the massive weights of an LLM, these methods transform time series patches into "pseudo-text" embeddings that the model can process.

In the TEST framework, Chenxi Sun introduces "text prototypes"—aligning time series embeddings with the LLM’s native text embedding space. This allows the model to treat numerical patterns as if they were semantic concepts. Similarly, $S^2$IP-LLM, developed by Zijie Pan and others, uses "semantic anchors" derived from word embeddings to provide contextually rich prompts. By mapping a specific temporal pattern to a semantic concept (e.g., "upward trend" or "volatile"), the model can apply its pre-trained knowledge of those concepts to the numerical task.

Time Series as Vision and Text

A surprising facet of this research is the use of non-linguistic foundation models. If a 1D signal can be mapped to a 2D space, it can be processed by powerful Vision Transformers.

  • Visual Inpainting: Mouxiang Chen demonstrates in VisionTS that converting time series into images allows models pre-trained on ImageNet to perform zero-shot forecasting. The model treats the future values of a time series as "missing pixels" to be filled in.
  • Pseudo-Images for EEG: In Large Transformers are Better EEG Learners, Bingxin Wang uses the AdaCT (Adapters for Converting Time series) method to transform brain signals into pseudo-images and text strings, bypassing the scarcity of labeled medical data.

These approaches suggest that the "reasoning" in foundation models is often a form of universal pattern matching that transcends the original modality of the training data.

Contextualization and Agentic Reasoning

Beyond simple forecasting, researchers are using LLMs as "agents" that can reason about the causes of numerical changes. Geon Lee and the team behind TimeCAP use dual LLM agents to contextualize events, bridging raw data with domain knowledge.

In specialized fields like meteorology, frameworks like Zephyrus by Sumanth Varambally give LLMs access to a Python "sandbox." This allows the model to write code to analyze high-dimensional meteorological arrays, grounding its linguistic reasoning in physical data. This shift from "model-as-predictor" to "model-as-analyst" is a key trend in making time series AI explainable.

The Risk of Pseudo-Alignment

Despite the excitement, the field faces a critical challenge: ensuring the alignment is meaningful. Liangwei Nathan Zheng warns in Understanding Why Large Language Models Can Be Ineffective that many current methods achieve only "pseudo-alignment."

Pseudo-alignment occurs when a model shifts the statistical distribution of time series data to match language embeddings without actually capturing the underlying structural relationships. In such cases, the LLM might perform no better than a simple linear model, or even a randomly initialized transformer. True alignment requires mapping the structural priors of time series—such as state transitions and causal dependencies—into the model’s reasoning space, as explored in SGCMA by Siming Sun.

As this research matures, the focus is shifting from simply "feeding numbers to LLMs" toward sophisticated, structure-aware translation layers that allow foundation models to "read" the physical world as fluently as they read text.

Papers in this version

Showing 60 of 210 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 140 papers
  2. Version 1 · 2026-10-04 · 210 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_representation_gap_modality__43bcba48,
  title = {cross-modal representation gap (modality mismatch)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-representation-gap-modality-mismatch/rdr_43bcba485ba9481e9fd3a51c9e137270},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_alignment_for_time_series_fo_43bcba48_v1,
  title = {Cross-Modal Alignment for Time Series Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-alignment-for-time-series-foundation-models/rdr_43bcba485ba9481e9fd3a51c9e137270/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}