Abstract
The recent rapid and significant increase of big data in our society has led to major impacts of machine learning and data mining technologies in various fields ranging from marketing to science. On the other hand, there still exist areas where only small-sized data are available for various reasons, for example, high data acquisition costs or the rarity of targets events. Machine learning tasks using such small data are usually difficult because of the lack of information available for training accurate prediction models. In particular, for long-term time-series prediction, the data size tends to be small because of the unavailability of the data between input and output times in training. Such limitations on the size of time-series data further make long-term prediction tasks quite difficult; in addition, the difficulty that the far future is more uncertain than the near future.
Introduction
Predicting the far future is inherently harder than predicting the next step because uncertainty accumulates over time. This challenge is compounded in "small data" scenarios—such as rare natural disasters, medical trials, or new business launches—where there aren't enough samples to train complex models effectively.
In standard long-term forecasting (predicting $x_{t+k}$ from $x_t$), the data points between $t$ and $t+k$ are often ignored during the training of a direct prediction model. This paper argues that this intermediate data contains valuable structural information about the trajectory of the series. The goal is to leverage this "middle-time" data during training to build a better model that only needs the current input at inference time.
Methodology: Generalized Distillation
The authors utilize the framework of Generalized Distillation, which combines traditional Knowledge Distillation with the "Learning Using Privileged Information" (LUPI) paradigm.
The process involves three main steps:
- The Teacher Model: A "Teacher" model is trained using Privileged Information ($x^*$). In this context, $x^*$ consists of the intermediate time steps (e.g., $x_{t+1}, x_{t+2}, \dots, x_{t+k-1}$). Because the teacher sees the "path" to the future, it can make more informed predictions than a model seeing only the start point.
- Generating Soft Targets: The trained teacher processes the training data and outputs "soft targets"—nuanced values representing its "knowledge" of the outcome.
- The Student Model: A "Student" model is trained to predict the target $x_{t+k}$ using only the standard input $x_t$. It minimizes a loss function balancing ground truth (hard targets) and the teacher’s nuanced predictions (soft targets).
By mimicking the teacher, the student learns to capture the underlying dynamics revealed by the intermediate steps without needing them during actual deployment.
Experimental Setup and Results
The researchers evaluated their method on three types of data: Synthetic Random Walk, Mackey-Glass chaotic time-series, and real-world Beijing PM2.5 air quality data.
Key Insights
- Intermediate Data as a Bridge: Historical datasets contain the "future" path. While unavailable at test time, this path can "teach" the model what a successful trajectory looks like.
- Proximity and Timing: In random walk and PM2.5 experiments, the authors found that intermediate data points closer to the input time (the "near future") were often more effective as privileged information.
- Complexity Gap: Distillation is most effective when the teacher's task is significantly more informative than the student's. If the task is "too easy" (as seen in Mackey-Glass results where accuracy was already 0.971), the benefit diminishes.
- Regularization Effect: The teacher’s soft targets act as intelligent regularization, preventing overfitting on small training sets.
Performance Summary
- Beijing PM2.5: The proposed method significantly outperformed the baseline as the input window size increased, helping the model navigate the "curse of dimensionality."
- Mackey-Glass: The method did not improve performance here because the baseline model was already near-optimal.
Conclusion and Future Work
Generalized distillation offers a promising path for long-term prediction in data-scarce environments by treating intermediate time steps as privileged information. Future work includes applying this to deep learning architectures like LSTMs or Transformers and developing automated methods for selecting the optimal intermediate steps and hyperparameters ($\lambda, T$) in small data settings.
Summary updated