TS-CLIP: Time Series Understanding by CLIP

Ziwen Chen, Xiaoyuan Zhang, Ming ZhuEMNLP 2025

TL;DR

This paper introduces TS-CLIP, a novel approach that adapts the powerful CLIP model for time series analysis. By introducing a "synonym bank" mechanism, TS-CLIP effectively addresses the challenge of label collapse in time series data, achieving state-of-the-art zero-shot performance on 51 datasets and paving the way for integrating time series into multimodal large language models.

What Problem Does This Paper Address?

Time series data, ubiquitous in fields like healthcare, finance, and meteorology, presents a unique challenge for natural language understanding. Unlike images or text, time series data often lacks rich semantic annotations and visual cues, making it difficult to align with language models. Existing methods struggle to bridge this gap, often focusing on time series forecasting rather than semantic understanding. A key issue is "label collapse," where models fail to differentiate between similar subclasses when trained with sparse or generalized textual descriptions. This paper aims to solve this by enabling robust semantic alignment between time series and natural language, thereby unlocking new possibilities for time series analysis and integration into multimodal AI systems.

Methods

The core innovation of TS-CLIP lies in its adaptation of the CLIP architecture. Instead of directly aligning time series with precise textual descriptions, TS-CLIP introduces a synonym bank. This mechanism leverages the word analogy capabilities of CLIP's text encoder to generate a distribution of potential synonym embeddings. This allows TS-CLIP to align time series with a broader semantic understanding rather than a single, potentially ambiguous, textual label.

Key components include:

  • TS Encoder: Based on the TimeMoE architecture, this encoder processes time series data.
  • CLIP Text Encoder: The pre-trained CLIP text encoder is used and kept frozen to leverage its extensive knowledge.
  • Synonym Bank: This is the novel component. It generates multiple near-synonymous embedding vectors by perturbing word embeddings. This helps to create a more robust alignment target and mitigate label collapse. The synonym generation process involves sampling from a Gaussian distribution with an anisotropic perturbation term to capture semantic relationships.
  • Contrastive Learning: TS-CLIP is trained using a contrastive loss, similar to CLIP, to maximize the similarity between aligned time series and text pairs while minimizing it for negative pairs.

Datasets

The paper utilizes the UCR Time Series Archive, a widely recognized benchmark for time series classification. For training and evaluation, they used 128 sub-datasets from this archive, covering diverse domains such as medical, energy, biological, and environmental data. The authors also mention that the complete training data will be publicly released.

Key Insights and Ideas

  • Synonym Bank for Robust Alignment: The synonym bank is crucial for overcoming the "label collapse" problem in time series. By providing a richer set of semantic targets, it allows for more nuanced alignment, especially when textual descriptions are sparse or ambiguous.
  • Leveraging CLIP's Word Analogy: The paper effectively demonstrates that CLIP's text encoder can perform word analogy, a capability that is instrumental in generating the synonym embeddings.
  • Bridging the Modality Gap: TS-CLIP successfully bridges the gap between time series and natural language, enabling semantic understanding of time series data. This is a significant step towards integrating time series into multimodal large language models (MLLMs).
  • Zero-Shot Performance: The model achieves impressive zero-shot performance, indicating its ability to generalize to unseen time series data without task-specific fine-tuning.

Key Results

  • State-of-the-Art Zero-Shot Performance: TS-CLIP achieved state-of-the-art (SOTA) performance in zero-shot settings on 51 out of 128 datasets from the UCR archive.
  • Effective Synonym Bank: Ablation studies showed that the synonym bank significantly outperforms using only subclass IDs for prompt generation, leading to much higher accuracy in zero-shot classification.
  • Few-Shot Generalization: TS-CLIP demonstrated strong few-shot generalization capabilities, surpassing supervised baselines with as few as 20 samples for fine-tuning.
  • Cross-Modal Alignment: Visualizations (t-SNE) confirmed that TS-CLIP effectively aligns time series samples with their corresponding class summaries in a shared embedding space.

Limitations and Future Work

  • Coarse-Grained Alignment: While TS-CLIP achieves general alignment, it may have limitations for tasks requiring very precise textual descriptions.
  • Data Scale: Creating a time series dataset comparable in scale to CLIP's (400 million samples) is challenging, making efficient data utilization a current limitation.
  • Semantic Conflicts: In real-world scenarios, highly similar waveform characteristics across different time series sources could lead to semantic conflicts, requiring further research into methods like prompt learning or one-shot learning for LLM integration.
  • Future Work: The authors suggest further research into efficiently integrating alignment models into LLMs for complex multimodal reasoning and exploring methods to address semantic conflicts in highly similar time series. They also highlight the potential for TS-CLIP to enable new downstream tasks like time series captioning and question answering.

Summary updated

BibTeX
@inproceedings{chen2025ts,
  author = {Ziwen Chen and Xiaoyuan Zhang and Ming Zhu},
  title = {{TS-CLIP: Time Series Understanding by CLIP}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure