An NCDE-based Framework for Universal Representation Learning of Time Series

Zihan Liu, Bowen Du, Junchen Ye, Xianqing Wen

Introduction

The quest for a "universal" representation in time series analysis—a single embedding capable of serving forecasting, classification, and imputation tasks—has been hindered by a fundamental trade-off. Contrastive learning (CL) methods typically excel at high-level classification by learning discriminative features but often fail at fine-grained forecasting. Conversely, generative methods focus on point-wise reconstruction, which benefits forecasting but may miss semantic abstractions necessary for classification. Furthermore, real-world data is frequently irregular and incomplete, posing a challenge for discrete models like Transformers or TCNs.

The CTRL Framework

To address these challenges, Liu et al. (IJCAI-24) introduced CTRL, an NCDE-based framework designed for task-generality and domain-adaptability.

Neural Controlled Differential Equations (NCDE) Backbone

Unlike traditional discrete models, CTRL employs NCDEs to model time series as continuous evolving processes. By fitting a continuous path $X(t)$ using natural cubic splines, the model can naturally handle missing values by evaluating the path at available timestamps. This continuous-time modeling provides a principled way to handle irregular sampling without the need for complex imputation pre-processing.

Dual-Task Self-Supervised Learning

CTRL bridges the gap between generative and discriminative learning through a dual-task self-supervised learning (SSL) strategy:

  1. Reconstruction Task: A linear head reconstructs the original time series from the embeddings, ensuring the capture of fine-grained temporal details.
  2. Hierarchical Contrastive Task: A contrastive loss is applied across multiple temporal scales to learn robust, high-level semantic features.
The overall architecture of CTRL, featuring the NCDE encoder and dual-task SSL branches for reconstruction and debiased contrasting.

Debiased Contrastive Learning

Standard contrastive learning often suffers from sampling bias. CTRL introduces two key mechanisms to mitigate this:

  • Hard Negative Construction (HNC): Synthesizes challenging negative samples through temporal shuffling and instance mixing, forcing the encoder to learn more nuanced representations.
  • False Negative Elimination (FNE): Identifies and removes samples from the negative pool that are semantically similar to the anchor, preventing the model from being penalized for recognizing inherent similarities in unlabeled data.

Experimental Results

The authors evaluated CTRL across 18 classification datasets, 4 forecasting datasets, and various imputation scenarios.

Universal Performance and Robustness

CTRL achieved the top average rank across all tasks compared to state-of-the-art methods like TS2Vec, CoST, and PatchTST. A standout feature of CTRL is its extreme robustness to missing data. As shown in the evaluation on the PowerCons and InsectWingbeatSound datasets, CTRL maintains high accuracy even as the missing data ratio approaches 90%, significantly outperforming competitors whose performance degrades rapidly.

Comparison of accuracy between CTRL and SOTA models across varying missing data rates, highlighting CTRL's superior robustness.

Synergy of Dual Tasks

Ablation studies confirmed that the combination of reconstruction and contrastive learning is essential. The reconstruction task provides the necessary detail for forecasting, while the contrastive task provides the discriminative power for classification. Together, they enable a truly universal representation.

Conclusion

CTRL represents a significant step toward general-purpose time series intelligence. By combining the continuous-time modeling of NCDEs with a balanced dual-task SSL approach, it overcomes the limitations of task-specific models and irregular data. Its state-of-the-art performance and parameter efficiency make it a robust foundation for future large-scale time series models.

Summary updated

BibTeX
@inproceedings{liu2024ncde,
  author = {Zihan Liu and Bowen Du and Junchen Ye and Xianqing Wen},
  title = {{An NCDE-based Framework for Universal Representation Learning of Time Series}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=G3Pl49LVsK},
}
Figure