In the landscape of machine learning, data is often likened to oil, but labeled data is more akin to refined gasoline: expensive, time-consuming to produce, and frequently in short supply. This scarcity is particularly acute in time series analysis. While sensors, medical devices, and financial systems generate a constant stream of raw data, identifying a specific heartbeat anomaly or an industrial equipment failure requires rare and expensive domain expertise. As Kexin Zhang et al. note in Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects, this "annotation bottleneck" prevents traditional supervised deep learning from scaling to high-stakes environments like healthcare or infrastructure monitoring.
To bypass this bottleneck, researchers are shifting away from training models from scratch on small, labeled datasets. Instead, they are developing methods that leverage the inherent structure of unlabeled data or repurpose the vast knowledge stored in models from other domains.
Reprogramming the Universal Computation Engine
One of the most surprising solutions to data scarcity is not to train a time series model at all, but to "reprogram" existing Large Language Models (LLMs). Tian Zhou et al. demonstrated in One Fits All: Power General Time Series Analysis by Pretrained LM that a frozen GPT-2 model can achieve state-of-the-art performance on time series tasks without changing its core weights.
The intuition behind this approach lies in the nature of self-attention. By treating time series patches as "tokens," the authors show that the LLM's attention mechanism acts as a universal computation engine, performing operations remarkably similar to Principal Component Analysis (PCA). This allows the model to capture complex temporal patterns even though it was never trained on a single sequence of sensor data. Similarly, Ming Jin et al. introduced TIME-LLM, which uses natural language prompts to provide context to these frozen models, significantly outperforming specialized models in zero-shot and few-shot scenarios where data is most limited.
This cross-domain transfer isn't limited to text. Chao-Han Huck Yang et al. proposed Voice2Series, which repurposes pre-trained acoustic models. Because speech is itself a complex time series, these models already possess powerful feature extractors for temporal data, which can be unlocked for general classification tasks with only a lightweight input transformation layer.
Learning from the Signal Itself
When pre-trained models aren't available, the alternative is to extract as much information as possible from unlabeled signals through self-supervised learning (SSL). The core idea is to create "pretext tasks" where the model must predict some hidden part of the data.
In medical contexts, where expert time is at a premium, Dani Kiyasseh and Tingting Zhu developed CLOCS. This framework leverages the unique spatial and temporal properties of ECG signals—such as the fact that different leads (views) of the same heart should yield consistent representations—to learn robust features without human labels. By contrasting these different views, the model learns the underlying physiology of the patient.
Other researchers focus on multi-domain consistency. Zhen Liu et al. proposed TS-TFC, which co-trains two networks—one looking at the time domain and another at the frequency domain. By forcing these two perspectives to agree on the same signal, the model generates high-quality "pseudo-labels" for unlabeled data, effectively multiplying the utility of the few expert labels that are available.
Synthetic Data and Sharpness-Awareness
If real-world labeled data is too expensive, can we simply invent it? Ege Onur Taga et al. introduced TimePFN, a model trained entirely on millions of synthetic examples generated by Gaussian Processes. By learning the general mathematical properties of multivariate time series from these simulations, the model can match the performance of models trained on full real-world datasets with as few as 500 fine-tuning points.
However, training on small datasets often leads to overfitting, where the model memorizes noise rather than learning patterns. To combat this, Jesus Barreda et al. developed COSCO, which uses Sharpness-Aware Minimization (SAM). Instead of just looking for the lowest error, SAM searches for "flat" regions in the loss landscape. The intuition is that if the model's weights change slightly and the error stays low, the model has likely found a generalizable pattern rather than a sharp, brittle peak of overfitted performance.
A Unified Path Forward
The shift toward label-efficient learning represents a fundamental change in how we approach time series. As summarized by Mohamed Ragab et al. in their review of label-efficient representation learning, the goal is to move away from task-specific silos. By combining self-supervised pre-training, cross-domain transfer from LLMs, and robust few-shot training techniques, the field is building foundation models that can understand the "language" of time series, regardless of whether that signal comes from a patient's heart, a factory sensor, or a stock market ticker.
Key Foundational Work
Unsupervised Feature Learning from Time Series (2016) — contributed 1 concept elements to this direction.
A review of unsupervised feature learning and deep learning for time-series modeling (2014) — contributed 1 concept elements to this direction.