Loss-Driven Adversarial Ensemble Deep Learning for On-Line Time Series Analysis

Hyungjin Ko, Jaewook Lee, Junyoung Byun, Bumho Son, Saerom ParkSustainability 2019 · OpenReview

TL;DR

This paper introduces a novel "Loss-Driven Adversarial Ensemble Deep Learning" (LADE) algorithm for robust and sustainable on-line time series analysis. LADE dynamically adjusts the weights of an ensemble of deep learning models based on their prediction loss, allowing it to adapt to changing data distributions and resist adversarial attacks, outperforming existing methods on various real-world datasets.

What Problem Does This Paper Address?

Analyzing time series data in real-world applications, especially in fields like finance or IoT, presents a significant challenge: data is often continuous and its underlying distribution can change over time (concept drift). Traditional deep learning models, trained offline, struggle to adapt to these shifts, leading to performance degradation. Ensemble methods, which combine multiple models, offer a path to improved robustness and performance. However, existing on-line ensemble methods often face issues with adapting to distribution changes or are computationally expensive to retrain. This paper aims to bridge this gap by proposing an on-line ensemble deep learning approach that efficiently adapts to evolving data characteristics without requiring full model retraining.

Methods

The core of the proposed method is an on-line ensemble deep learning algorithm that dynamically adjusts the weights of its constituent deep learning models. Instead of fixed weights or simple averaging, LADE uses a loss-driven adversarial approach.

Here's a breakdown:

  • Candidate Models: The ensemble is composed of multiple deep learning models (e.g., multilayer perceptrons) trained offline on an initial dataset.
  • On-line Weight Adjustment: As new data arrives in a streaming fashion, the algorithm calculates the loss of each base model on this new data. The ensemble weights are then updated based on these losses. Models with higher losses receive lower weights, effectively downplaying their influence.
  • Adversarial Regret Measure: The authors introduce a novel regret measure that considers an adversarial assumption, aiming to minimize the total cumulative loss under worst-case scenarios. This theoretical underpinning helps to prove the convergence of the ensemble weights.
  • Framework: The paper outlines a general framework that includes an off-line phase for initial model training and an on-line phase for continuous adaptation and prediction. The on-line phase involves processing incoming data in mini-batches to update ensemble weights.

What's new here is the combination of:

  1. Deep learning models as base learners: Leveraging the power of deep learning for individual predictions.
  2. On-line weight adaptation based on loss: A direct mechanism to respond to model performance degradation.
  3. Adversarial assumption in regret: A theoretical framework to ensure robust convergence.

Datasets

The proposed method was evaluated on a variety of datasets:

  • Simulated Data:
    • Simple sine function time series.
  • Composite sine function time series (combination of three sine functions with different frequencies).
  • Financial Time Series Data:
    • S&P 500 (5-min intervals).
  • Nasdaq future (daily intervals).
  • Gold future (daily intervals).
  • Sugar future (daily intervals).
  • Bitcoin (daily intervals).
  • Non-Financial Time Series Data:
    • Temperature (Austin, Texas, daily intervals).
  • Power consumption (American Electric Power, hourly data aggregated to daily).

The paper does not introduce new benchmark datasets but uses publicly available real-world financial and non-financial time series data, along with synthetically generated data to test specific properties.

Key Insights and Ideas

  • Adaptability is Key: The most significant insight is that for time series, especially those with changing distributions, adaptive ensemble weighting is crucial. Simply averaging models or using fixed weights is insufficient.
  • Loss as a Direct Signal: Using the prediction loss of individual models as the direct signal for weight adjustment is an intuitive and effective way to penalize underperforming models in real-time.
  • Robustness to Attacks: The method demonstrates remarkable resilience to intentional attacks where base classifiers are manipulated. The ensemble can effectively down-weight or exclude attacked models, maintaining overall system performance.
  • Sustainability in Distribution Shifts: The algorithm shows a strong ability to adapt to changes in data distribution, both in terms of input features and target variable imbalance, by dynamically reallocating weights.
  • Theoretical Guarantees: The introduction of an adversarial regret measure provides a theoretical foundation for the convergence of ensemble weights, ensuring that the algorithm aims for an optimal solution.

Key Results

  • Outperformance: The proposed Loss-Driven Adversarial Ensemble Deep Learning (LADE) method consistently outperformed baseline ensemble methods (e.g., equal weight, random forest, gradient boosting) across simulated, financial, and non-financial time series datasets in terms of accuracy, precision, recall, and AUC.
  • Accuracy Improvements: For financial datasets, LADE achieved an average accuracy difference of up to 46.69% over a trivial classifier. For non-financial data, it substantially outperformed other algorithms on power consumption data.
  • Robustness to Attacks: In experiments with up to 5 attacked classifiers (out of 12), the ensemble accuracy degraded only slightly. Performance dropped significantly only when 6 or more classifiers were attacked, but even then, accuracy recovered to some extent.
  • Adaptability to Distribution Changes: The weight distribution of the ensemble models visibly shifted over time in response to changes in data distribution and target imbalance, demonstrating the algorithm's adaptive nature.
  • Hyperparameter Sensitivity: The hyperparameter η, controlling the learning rate for weight updates, was found to be crucial, with a significant increase in accuracy observed when η was increased from 1.0 to 10.0.

Limitations and Future Work

  • Classification Focus: The current study focuses on binary classification tasks. The authors suggest that the algorithm can be extended to regression problems by using appropriate loss functions.
  • Deep Learning Architectures: While multilayer perceptrons were used as base classifiers, the authors propose that incorporating sequence-aware models like Recurrent Neural Networks (RNNs) could further improve prediction performance, especially for time series that inherently depend on sequential data.
  • Unsupervised Learning: The authors aim to extend the algorithm to unsupervised problems, such as anomaly detection, by developing suitable loss functions.
  • Computational Cost: While the method avoids full model retraining, the on-line adaptation still involves processing incoming data and updating weights, which could be a consideration for extremely high-velocity data streams.
  • Attack Strength: The authors note that their adversarial attack scenario was "strong," leading to immediate disappearance of attacked model weights. Future work could explore more nuanced attack scenarios.

Summary updated

Abstract

Developing a robust and sustainable system is an important problem in which deep learning models are used in real-world applications. Ensemble methods combine diverse models to improve performance and achieve robustness. The analysis of time series data requires dealing with continuously incoming instances; however, most ensemble models suffer when adapting to a change in data distribution. Therefore, we propose an on-line ensemble deep learning algorithm that aggregates deep learning models and adjusts the ensemble weight based on loss value in this study. We theoretically demonstrate that the ensemble weight converges to the limiting distribution, and, thus, minimizes the average total loss from a new regret measure based on adversarial assumption. We also present an overall framework that can be applied to analyze time series. In the experiments, we focused on the on-line phase, in which the ensemble models predict the binary class for the simulated data and the financial and non-financial real data. The proposed method outperformed other ensemble approaches. Moreover, our method was not only robust to the intentional attacks but also sustainable in data distribution changes. In the future, our algorithm can be extended to regression and multiclass classification problems.
BibTeX
@inproceedings{ko2019loss,
  author = {Hyungjin Ko and Jaewook Lee and Junyoung Byun and Bumho Son and Saerom Park},
  title = {{Loss-Driven Adversarial Ensemble Deep Learning for On-Line Time Series Analysis}},
  booktitle = {Sustainability},
  year = {2019},
  url = {https://openreview.net/forum?id=eC0EFyxDDG},
}
Figure