An NCDE-based Framework for Universal Representation Learning of Time Series

Zihan Liu, Bowen Du, Junchen Ye, Xianqing Wen, Leilei Sun

Abstract

Exploiting self-supervised learning (SSL) to extract the universal representations of time series could not only capture the natural properties of time series but also offer huge help to the downstream tasks. Nevertheless, existing time series representation learning (TSRL) methods face challenges in attaining universality. Indeed, existing methods relying solely on one SSL strategy (either contrastive learning (CL) or generative) often fall short in capturing rich semantic information for various downstream tasks. Moreover, time series exhibit diverse distributions and inherent characteristics, particularly with the common occurrence of missing values, posing a notable challenge for existing backbones in effectively handling such diverse time series data. To bridge these gaps, we propose CTRL, a framework for universal TSRL. For the first time, we employ Neural Controlled Differential Equation (NCDE) as the backbone for TSRL, which captures the continuous processes and exhibits robustness to missing data. Additionally, a dual-task SSL strategy, integrating both reconstruction and contrasting tasks, is proposed to enrich the semantic information of the learned representations. Furthermore, novel hard negative construction and false negative elimination mechanisms are proposed to improve sampling efficiency and reduce sampling bias in CL. Finally, extensive experiments demonstrate the superiority of CTRL in forecasting, classification, and imputation tasks, particularly its outstanding robustness to missing data.

Introduction

Time series data is ubiquitous across domains such as healthcare, finance, and environmental monitoring. Despite its prevalence, developing a universal representation learning framework that performs consistently across diverse downstream tasks—such as forecasting, classification, and imputation—remains a significant challenge. Current research identifies two primary obstacles: the Task-Generality Gap and the Domain-Adaptability Gap.

The Task-Generality Gap arises because existing self-supervised learning (SSL) methods typically favor either contrastive or generative approaches. Contrastive methods excel at capturing high-level semantic features for classification but often struggle with the fine-grained details required for forecasting. Conversely, generative (reconstruction-based) methods are effective for forecasting but lack the discriminative power needed for classification. The Domain-Adaptability Gap refers to the difficulty discrete-time backbones (like Transformers or TCNs) face when dealing with real-world, "messy" data characterized by irregular sampling and missing values.

Methodology: The CTRL Framework

To address these challenges, the researchers introduced CTRL, a framework built on three core technical innovations.

1. NCDE as a Continuous Backbone

Unlike traditional models that treat time series as a sequence of discrete tokens, CTRL utilizes Neural Controlled Differential Equations (NCDEs). NCDEs model the underlying continuous path of a time series, allowing the framework to naturally handle missing data through interpolation of the continuous process. This shift from discrete to continuous modeling is fundamental to CTRL's robustness.

2. Dual-Task SSL Strategy

CTRL employs a hybrid SSL strategy that integrates both reconstruction and hierarchical contrasting. The reconstruction task ensures the model captures point-wise details necessary for forecasting and imputation, while the hierarchical contrastive task extracts high-level features for classification.

The CTRL architecture featuring NCDE encoders and a dual-task SSL strategy.

3. Debiased Contrasting

To improve the efficiency of contrastive learning, CTRL introduces Hard Negative Construction and False Negative Elimination. By synthesizing hard negatives through mixing and shuffling, and filtering out false negatives using similarity thresholds, the model avoids the sampling biases common in periodic time series data.

Experimental Results

The performance of CTRL was evaluated across 26 classification datasets, 4 forecasting benchmarks, and multiple imputation tasks.

Performance and Robustness

CTRL achieved the top average rank across all categories. In forecasting, it outperformed specialized models like PatchTST. In imputation, it showed a 32.3% average MSE improvement over existing state-of-the-art methods. Most notably, CTRL demonstrated exceptional resilience to missing data. As shown in the figure below, while the accuracy of competing models like TS2Vec and TST plummeted as missing rates increased, CTRL maintained high performance even with 50% to 90% missing data.

Performance comparison of CTRL against other methods under varying missing data rates on PowerCons and InsectWingbeatSound datasets.

Conclusion

CTRL represents a significant advancement in universal time series representation learning. By leveraging the continuous nature of NCDEs and a dual-task SSL strategy, it bridges the gap between different downstream tasks and handles real-world data irregularities with unprecedented robustness. This framework serves as a foundational step toward the development of large-scale time series foundational models.

Summary updated

Abstract

Exploiting self-supervised learning (SSL) to extract the universal representations of time series could not only capture the natural properties of time series but also offer huge help to the downstream tasks. Nevertheless, existing time series representation learning (TSRL) methods face challenges in attaining universality. Indeed, existing methods relying solely on one SSL strategy (either contrastive learning (CL) or generative) often fall short in capturing rich semantic information for various downstream tasks. Moreover, time series exhibit diverse distributions and inherent characteristics, particularly with the common occurrence of missing values, posing a notable challenge for existing backbones in effectively handling such diverse time series data. To bridge these gaps, we propose CTRL, a framework for universal TSRL. For the first time, we employ Neural Controlled Differential Equation (NCDE) as the backbone for TSRL, which captures the continuous processes and exhibits robustness to missing data. Additionally, a dual-task SSL strategy, integrating both reconstruction and contrasting tasks, is proposed to enrich the semantic information of the learned representations. Furthermore, novel hard negative construction and false negative elimination mechanisms are proposed to improve sampling efficiency and reduce sampling bias in CL. Finally, extensive experiments demonstrate the superiority of CTRL in forecasting, classification, and imputation tasks, particularly its outstanding robustness to missing data.
BibTeX
@inproceedings{liu2024ncde,
  author = {Zihan Liu and Bowen Du and Junchen Ye and Xianqing Wen and Leilei Sun},
  title = {{An NCDE-based Framework for Universal Representation Learning of Time Series}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=G3Pl49LVsK},
}
Figure