Introduction
In the era of massive foundation models, the computational and storage costs associated with training on large-scale datasets have become a significant bottleneck. Dataset Condensation (DC) has emerged as a promising solution, aiming to distill a large-scale dataset into a small, synthetic version that preserves the essential knowledge required for model training. While DC has seen substantial success in image classification, its application to Time Series Forecasting (TSF) has historically lagged behind.
Existing DC methods often fail to capture the rigorous point-wise accuracy required for forecasting. In their NeurIPS 2024 paper, researchers from Shanghai Jiao Tong University introduce CondTSF, a lightweight, "one-line" plugin designed to bridge the gap between classification-centric condensation and the specific requirements of time series data.
The Challenge: Why Classification DC Fails TSF
Most dataset condensation techniques, such as trajectory matching or gradient matching, were originally designed for classification tasks. In classification, the model's objective is to identify the correct category; the exact probability distribution (logits) is secondary as long as the correct class is predicted.
Time Series Forecasting is fundamentally different. It requires high-fidelity trajectory prediction where even small shifts in value can lead to significant errors. Standard DC methods typically focus on matching the parameters of a model trained on synthetic data to those of a model trained on real data (the "Gradient Term"). However, the authors demonstrate that for TSF, matching gradients is insufficient. One must also ensure that the output values (the labels) are synchronized with the input features to maintain the structural integrity of the time series.
CondTSF: Synchronizing Gradients and Values
The researchers provide a theoretical decomposition of the TSF condensation objective into two distinct, optimizable components:
- The Gradient Term: This ensures the model's sensitivity to inputs is aligned, a task handled by existing DC backbones.
- The Value Term: This ensures the actual predicted values are accurate and anchored to the input trajectory.
To address the Value Term, the authors propose CondTSF. Unlike complex optimization loops, CondTSF utilizes a simple additive update rule. It leverages an "expert model"—a model pre-trained on the full dataset—to predict outcomes for the current synthetic inputs. The synthetic labels are then nudged toward these expert predictions using a moving average:
$$s_{label}^{(i+1)} = (1 - \beta) \cdot s_{label}^{(i)} + \beta \cdot \text{Expert}(\text{synthetic\_input})$$
This approach is computationally efficient and can be integrated into nearly any existing DC backbone (such as MTT, DC, or TESLA) with a single line of code. By updating labels in tandem with feature optimization, CondTSF ensures that the synthetic dataset remains internally consistent for regression-based tasks.
Experimental Evaluation
The efficacy of CondTSF was evaluated across eight standard time series benchmarks, including Electricity, Traffic, Weather, ExchangeRate, and four variants of the ETT (Electricity Transformer Temperature) datasets. These datasets represent a wide range of sampling frequencies and domain characteristics.
Experiments were conducted under extreme "distill ratios," in some cases compressing datasets to as little as 0.83‰ of their original size. To facilitate the condensation process, the authors utilized DLinear—a simple linear model—as the "expert." Linear models were chosen for their stable gradients, which provide a reliable signal for distilling features that generalize to more complex architectures.
Key Results and Insights
- Significant Error Reduction: CondTSF consistently outperformed baseline DC methods across all datasets. On the ExchangeRate dataset using the MTT backbone, CondTSF reduced the Mean Absolute Error (MAE) from 0.778 to 0.195, representing a 75% improvement. In several benchmarks, error reduction exceeded 90%.
- Cross-Architecture Generalization: A critical finding was that synthetic data distilled using a simple linear expert (DLinear) effectively trained much more complex models, including MLPs, LSTMs, and CNNs. This suggests that CondTSF captures fundamental temporal patterns rather than model-specific artifacts.
- Resilience at Low Ratios: Previous DC methods often collapse when the condensation ratio is extremely low. CondTSF remains robust in these scenarios by providing a constant "value anchor" through the expert model, preventing the synthetic labels from drifting during optimization.
Limitations and Future Outlook
While CondTSF shows remarkable empirical success, the authors note that their theoretical proofs primarily rely on the properties of linear models. While the method generalizes well to non-linear architectures, a more rigorous theoretical framework for Transformers remains an area for future study. Additionally, the method requires a pre-trained expert model, which is a common but necessary prerequisite for trajectory-matching approaches.
As the field moves toward Time Series Foundation Models, CondTSF offers a scalable path for creating high-quality, compact datasets for efficient fine-tuning and model alignment, potentially reducing the massive computational overhead currently associated with large-scale temporal modeling.
Summary updated