LLMs Meet Cross-Modal Time Series Analytics: Overview and Directions

Introduction

Large Language Models (LLMs) have fundamentally changed natural language processing, yet their application to numerical time series data remains a significant challenge. While both text and time series are sequential, LLMs are "text-native," trained on massive corpora of human language rather than numerical sensor data. This creates a cross-modality gap, where standard models struggle to inherently understand the patterns, frequencies, or scaling of numerical sequences. This paper provides a comprehensive roadmap for bridging this gap through systematic integration of textual knowledge (context, metadata, instructions) with structured time series data to improve tasks like forecasting, anomaly detection, and classification.

Methodology: Bridging the Cross-Modality Gap

The core contribution of this work is a formal taxonomy categorizing how researchers bridge the gap between text-trained models and numerical data. These strategies are divided into three primary approaches:

1. Modality Conversion

This approach transforms numerical time series into a textual format that an LLM can "read" directly. Techniques include serializing numbers into strings (e.g., "1.2, 3.4, 5.6") or framing forecasting as a sentence-to-sentence translation task, known as "PromptCast." This is particularly effective for zero-shot scenarios, leveraging the LLM's pre-existing reasoning without any additional training on specific time series data.

2. Cross-Modal Alignment

Alignment focuses on making the latent representations of time series and text similar within the model's embedding space, ensuring the model understands that a specific "spike" in a graph corresponds to a specific "event" described in text.

• Retrieval: Using time series snippets to find relevant textual information in external databases.
• Contrastive Learning: Training the model to maximize similarity between corresponding text-time series pairs while minimizing it for mismatched pairs.
• Knowledge Distillation (KD): Transferring the reasoning capabilities of a large "teacher" LLM into a more efficient, task-specific "student" architecture.

3. Cross-Modal Fusion

Fusion involves merging modalities into a unified representation. This can occur at various stages—early (raw features), middle (embeddings), or late (predictions). Methods include simple addition of feature vectors, which is compact, or concatenation, which is more complex but preserves more information.

Expand

Source, p. 2

Key Insights and Results

The fundamental reason LLMs are effective for time series is the shared sequential nature of both data types. The logic and pattern-recognition capabilities learned from language appear transferable to numerical sequences. Key results from recent literature highlight:

• Transferability: LLMs benefit from large-scale self-supervised pre-training, allowing them to adapt to new tasks faster than traditional RNNs or CNNs.
• Interpretability: LLM-based methods can provide not just a forecast, but a natural language explanation for why that forecast was made.
• Efficiency vs. Effectiveness: While powerful, LLMs can be "overkill." Smaller, task-specific models sometimes outperform massive LLMs, which may overfit or "hallucinate" patterns in numerical data.

These strategies are applied across various domains, including finance analysis, anomaly detection, traffic forecasting, and disease classification using benchmarks like Time-MMD, a multi-domain multimodal dataset.

Challenges and Future Directions

Despite their potential, several "open issues" remain:

• Computational Cost: Processing long multivariate time series through an LLM's attention mechanism is extremely expensive.
• Transparency: LLMs remain "black boxes," necessitating models that can expose internal reasoning for high-stakes fields like healthcare.
• Hallucinations: LLMs can generate plausible-looking but mathematically incorrect numerical outputs.

Future research is pointing toward Multimodal LLMs that integrate images and sensors, Multi-Agent Systems where different LLMs collaborate, and Retrieval-Augmented Generation (RAG) to pull in real-time historical data.

@inproceedings{liu2025llms,
  author = {Chenxi Liu and Hao Miao and Cheng Long and Yan Zhao and Ziyue Li and Panos Kalnis},
  title = {{LLMs Meet Cross-Modal Time Series Analytics: Overview and Directions}},
  booktitle = {SSTD},
  year = {2025},
  doi = {10.1145/3748777.3748810},
}