PACOL: Poisoning Attacks Against Continual Learners

Huayu Li, Gregory DitzlerarXiv 2023 · OpenReview

TL;DR

This paper introduces PACOL, a novel data poisoning attack specifically designed for continual learning systems. PACOL can cause catastrophic forgetting by subtly perturbing data without changing labels, making it stealthier than traditional label-flipping attacks. The research highlights the vulnerability of common continual learning methods to such attacks and evaluates defense mechanisms.

What Problem Does This Paper Address?

Continual learning systems are designed to learn sequentially from streaming data, adapting to new information without forgetting past knowledge. However, these systems are often exposed to untrusted data sources, making them susceptible to adversarial attacks. Existing data poisoning attacks primarily focus on single-task learning or have limitations in their stealthiness. This paper addresses the gap of sophisticated, stealthy poisoning attacks that specifically target the catastrophic forgetting phenomenon in continual learning, aiming to degrade performance on previously learned tasks.

Methods

The paper proposes PACOL (Poisoning Attacks Against Continual Learners), a novel data poisoning attack. PACOL builds upon the idea of label-flipping attacks but introduces a key innovation: it perturbs input data points (e.g., images) within a small, bounded region (l∞-norm) such that their original labels remain unchanged. The core idea is to generate adversarial samples that, when introduced during training, cause the model's gradients to align with those of a label-flipped version of the same sample. This alignment, when applied to a target task, forces the model to "forget" its knowledge of that task.

The PACOL attack is formulated as an optimization problem that minimizes the distance between the gradients of the model on a clean sample and its perturbed counterpart. The paper explores two distance metrics: Euclidean (l2) distance and negative cosine similarity. The attack is then integrated into the training loop of continual learning algorithms.

Datasets

The paper evaluates its proposed attack and defense mechanisms on several standard continual learning benchmarks:

  • Rotation MNIST (R-MNIST): A domain incremental learning dataset with five tasks, each involving a different rotation of the original MNIST digits.
  • Split MNIST (S-MNIST): A task incremental learning dataset where each of the five tasks involves a binary classification problem, distinguishing between pairs of digits.
  • Split SVHN (S-SVHN): Similar to S-MNIST, but using images from the Street View House Numbers dataset.
  • Split CIFAR-10 (S-CIFAR): A task incremental learning dataset split into five binary classification tasks, using images from the CIFAR-10 dataset.

Key Insights and Ideas

  • Stealthy Catastrophic Forgetting: PACOL demonstrates that subtle perturbations to input data, without altering labels, can effectively induce catastrophic forgetting in continual learning models. This is a significant advancement over label-flipping attacks, which are more detectable.
  • Vulnerability of Continual Learning: The research highlights that even sophisticated continual learning methods like generative replay (DGR) and regularization-based approaches (EWC, Online EWC, SI) are susceptible to PACOL attacks.
  • Defense Challenges: While defenses like l2-norm outlier detection, One-Class SVM, and Isolation Forest can mitigate label-flipping attacks, they struggle to effectively detect and remove the imperceptible poisoned samples generated by PACOL, especially when using deep feature embeddings.
  • Targeted vs. Non-Targeted Poisoning: The attack focuses on poisoning non-targeted tasks to induce forgetting in a targeted task, showcasing a strategic approach to manipulation.

Key Results

  • Performance Degradation: PACOL attacks consistently led to significant performance degradation on the targeted tasks across various continual learning algorithms and datasets. For instance, in Table 1, label-flipping attacks often caused a substantial drop in accuracy, and PACOL's stealthier approach also achieved similar or greater degradation.
  • Defense Effectiveness Varies: Defense methods showed varying success rates. L2-norm outlier defense was effective against label-flipping attacks, especially with raw data or deep features, but struggled with PACOL's imperceptible perturbations. One-Class SVM showed some promise against PACOL with deep features on S-CIFAR, while Isolation Forest and Local Outlier Factor were generally less effective.
  • White-box Advantage: White-box PACOL attacks generally resulted in the largest performance drops, as expected, due to the adversary's full knowledge of the model. However, gray-box and black-box attacks also demonstrated considerable effectiveness.
  • Generalizability: The PACOL attack was shown to be effective across different datasets (MNIST variants, S-SVHN, S-CIFAR) and different types of continual learning algorithms (regularization-based and generative replay).

Limitations and Future Work

Limitations:

  • Human Censorship vs. Imperceptible Perturbations: The paper notes that while label-flipping attacks are more detectable by humans, PACOL's imperceptible perturbations are harder to detect even with advanced defense mechanisms.
  • Specific Attack Types: The research focuses on specific types of poisoning attacks (label-flipping and PACOL) and doesn't cover all possible adversarial strategies.
  • Defense Limitations: Current defense methods are shown to be insufficient against stealthy poisoning attacks like PACOL, especially in complex scenarios.

Future Work:

  • Developing Robust Defenses: The authors emphasize the need for more robust defense strategies that can effectively handle imperceptible poisoning attacks in continual learning.
  • Algorithmic and Architectural Strategies: Future work will explore developing data-agnostic, algorithmic, and architectural-level strategies to ensure adversarial robustness in continual machine learning.
  • Formal Benchmarks: Establishing formal benchmarks for adversaries in continual and lifelong learning environments is identified as a critical area for future investigation.
  • Ethical Implications: The paper acknowledges the ethical implications of adversarial attacks, particularly in targeting forgetting in ways that could violate fairness, and urges caution in the application of these techniques.

Summary updated

Abstract

Continual learning algorithms are typically exposed to untrusted sources that contain training data inserted by adversaries and bad actors. An adversary can insert a small number of poisoned samples, such as mislabeled samples from previously learned tasks, or intentional adversarial perturbed samples, into the training datasets, which can drastically reduce the model's performance. In this work, we demonstrate that continual learning systems can be manipulated by malicious misinformation and present a new category of data poisoning attacks specific for continual learners, which we refer to as {\em Poisoning Attacks Against Continual Learners} (PACOL). The effectiveness of labeling flipping attacks inspires PACOL; however, PACOL produces attack samples that do not change the sample's label and produce an attack that causes catastrophic forgetting. A comprehensive set of experiments shows the vulnerability of commonly used generative replay and regularization-based continual learning approaches against attack methods. We evaluate the ability of label-flipping and a new adversarial poison attack, namely PACOL proposed in this work, to force the continual learning system to forget the knowledge of a learned task(s). More specifically, we compared the performance degradation of continual learning systems trained on benchmark data streams with and without poisoning attacks. Moreover, we discuss the stealthiness of the attacks in which we test the success rate of data sanitization defense and other outlier detection-based defenses for filtering out adversarial samples.
BibTeX
@misc{li2023pacol,
  author = {Huayu Li and Gregory Ditzler},
  title = {{PACOL: Poisoning Attacks Against Continual Learners}},
  year = {2023},
  eprint = {2311.10919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.10919},
}
Figure