TL;DR
This paper introduces a novel Multi-Task Optimization based Co-training (MTO-CT) framework that leverages shared knowledge between related prediction tasks to improve accuracy. By co-training Long Short-Term Memory (LSTM) models for electricity consumption prediction, MTO-CT demonstrates significant performance gains over independent task training, especially when dealing with multiple prediction horizons or geographical locations.
What Problem Does This Paper Address?
Predicting electricity consumption is crucial for grid management, but it often involves tackling numerous related tasks simultaneously. These tasks might include predicting consumption at different time steps ahead (e.g., 1-hour, 6-hour, 24-hour forecasts) or across different geographical regions. Traditionally, each of these prediction tasks is solved in isolation, ignoring the potential for shared learning. This approach misses out on valuable common knowledge that could be extracted and transferred between tasks, thereby improving the overall prediction accuracy and efficiency for all involved tasks. The paper aims to bridge this gap by developing a framework that explicitly facilitates knowledge sharing among these related prediction problems.
Methods
The core of the proposed solution is the Multi-Task Optimization based Co-training (MTO-CT) framework. This framework employs Long Short-Term Memory (LSTM) networks as the underlying prediction models. The "knowledge" within these models is represented by their connection weights and biases.
The key innovation lies in the inter-task knowledge transfer module. This module orchestrates the sharing of learned information between tasks. For a given "target" task, it first identifies potentially helpful "source" tasks. This selection process uses:
- Probability Matching (PM): To calculate the likelihood that a source task can contribute positively to the target task, based on historical performance.
- Stochastic Universal Selection (SUS): To probabilistically select the most beneficial source tasks from the pool of candidates.
Once source tasks are selected, their knowledge is reused in the target task through evolutionary operations, specifically:
- Mutation: Inspired by Differential Evolution (DE), a mutant solution is generated using the parameters from selected source tasks.
- Crossover: Also from DE, this operation combines the mutant solution with the target task's current parameters to create a new candidate solution.
The framework iteratively refines the models, updating task selection probabilities based on their observed helpfulness. This process is framed as a co-training approach where tasks learn from each other within a unified optimization paradigm.
Datasets
The authors utilized electricity consumption data downloaded from the Australian Energy Market Operator (AEMO). This data covers 30-minute intervals from November 2020 to November 2021 for five Australian states: VIC, NSW, SA, QLD, and TAS.
Two sets of prediction tasks were created from this data:
- Set A: One-step ahead prediction for each of the five states (total of 5 tasks).
- Set B: Multi-step ahead prediction (6-step, 12-step, 18-step, and 24-step ahead) for each of the five states (total of 20 tasks).
The data was normalized to [0, 1] using min-max normalization. For both sets, a time window of 24 timestamps was used as input to predict future values. The dataset was split into 80% for training and 20% for testing.
Key Insights and Ideas
- The Power of Shared Knowledge: The fundamental insight is that related prediction tasks, even those with slightly different objectives (like different prediction horizons), share underlying patterns. Explicitly enabling knowledge transfer between these tasks can significantly boost performance beyond what's achievable by training each task independently.
- Intelligent Source Task Selection: Simply transferring knowledge from all other tasks is inefficient. The paper's approach of using probability matching and stochastic universal selection to identify the most helpful source tasks is crucial for effective knowledge transfer.
- Evolutionary Operations for Knowledge Reuse: Employing mutation and crossover from evolutionary algorithms provides a robust mechanism to integrate knowledge from source tasks into the target task's model parameters. This is more sophisticated than simple averaging or direct parameter copying.
- LSTM as a Knowledge Representation: The choice of LSTM is well-suited because its internal states and weights naturally encode temporal dependencies and learned patterns, making them amenable to transfer.
Key Results
The MTO-CT framework consistently outperformed the Single-Task Prediction (STP) baseline across both task sets.
- Set A (5 tasks, one-step ahead): MTO-CT showed superior performance in both training and testing RMSE for all five tasks. The paper states MTO-CT achieved a 5/0/0 advantage (wins/ties/losses) over STP for both training and testing.
- Set B (20 tasks, multi-step ahead): MTO-CT also demonstrated significant improvements. For the training set, MTO-CT outperformed STP on 14 out of 20 tasks. For the testing set, MTO-CT led to better performance on 17 out of 20 tasks.
- Statistical Significance: Wilcoxon signed-rank tests at the 0.05 level were used to confirm the statistical significance of the observed differences, with better results often highlighted in bold.
Limitations and Future Work
- Limitations:
- The study primarily focuses on electricity consumption prediction, and its generalizability to other domains with different data characteristics might require further investigation.
- While the paper demonstrates superiority, some specific tasks within Set B (multi-step ahead) showed worse performance with MTO-CT compared to STP. This suggests that the effectiveness of knowledge transfer can be task-dependent.
- Future Work:
- Enriching Input Data: Incorporating additional relevant time series data, such as temperature, could further enhance prediction accuracy.
- Exploring Other Models: Evaluating MTO-CT with different LSTM variants or entirely different prediction models (e.g., [27]) could broaden its applicability.
- Broader Applications: Applying the MTO-CT framework to other domains where multi-task learning is beneficial, such as graph matching [28], feature extraction [29], and service composition [30], is a promising avenue.
Summary updated