Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization

Yuhao He, Jinyu Tian, Xianwei Zheng, Li Dong, Yuanman Li, Leo Yu Zhang, Jiantao ZhouarXiv 2024 · DOI · arXiv

Introduction

Recent research from the Macau University of Science and Technology introduces the Deferred Poisoning Attack (DPA), a novel category of adversarial threats targeting machine learning models. Unlike traditional data poisoning, which typically degrades model performance during the training or validation phases, DPA acts as a "sleeper cell." The poisoned model maintains high accuracy in controlled environments but fails catastrophically when exposed to real-world conditions.

Overview of the Deferred Poisoning Attack (DPA) lifecycle, showing how a poisoned model remains stealthy until deployment.

The Stealth Gap in Data Poisoning

Most existing poisoning attacks are easily detectable because they cause visible drops in validation accuracy or specific misclassifications during testing. DPA addresses this "loudness" by ensuring the poisoned model is indistinguishable from a clean one during standard evaluation. The attack's toxicity is deferred until the model encounters slight variations in the wild—such as natural noise or subtle adversarial perturbations—making it particularly dangerous for safety-critical applications like autonomous driving.

Methodology: Hessian Singularization

The technical core of DPA is Hessian Singularization. In deep learning, the Hessian matrix represents the local curvature of the loss function. A robust model typically converges to a "flat" minimum with low curvature. Conversely, DPA forces the model into a "sharp" minimum characterized by high local curvature.

Comparison of loss landscapes for clean and poisoned models, illustrating how Hessian singularization creates extreme sensitivity to input perturbations.

To achieve this, the authors introduce a regularization term, $Q$, targeting the trace of the Hessian matrix. The optimization follows a two-stage bilevel process:

  1. Model Training: The model is optimized to minimize standard cross-entropy loss on both clean and poisoned data to maintain high accuracy.
  2. Perturbation Update: Poisoning noise is iteratively updated to maximize local curvature (singularization) around training samples.

To manage the computational complexity of calculating the full Hessian ($O(P^2)$), the researchers employ the Hessian-Vector Product (HVP), reducing complexity to $O(P)$ and enabling the attack on modern, deep neural networks.

Experimental Validation

The effectiveness of DPA was validated across CIFAR-10, SVHN, and TinyImageNet datasets using architectures such as VGG16, ResNet, and DenseNet.

Key Findings

  • Stealthiness: On TinyImageNet, a poisoned ResNet18 achieved a validation accuracy of 72%, matching the performance of a clean model.
  • Fragility: Despite high accuracy, robustness plummeted. For VGG16 on CIFAR-10, the minimal perturbation required to cause misclassification decreased by approximately 70%.
  • Natural Noise Sensitivity: Poisoned models proved highly vulnerable to common environmental noise (e.g., Gaussian, Poisson, or Rayleigh noise). This suggests that a DPA-poisoned system might function perfectly in clear weather but fail under light rain or fog.
  • Transferability: Perturbations generated for one architecture (e.g., VGG16) successfully poisoned others (e.g., ResNet), indicating a potent black-box threat.

Conclusion and Defensive Outlook

DPA represents a significant shift in poisoning strategies, prioritizing stealth and environmental triggers over immediate performance degradation. While traditional adversarial training offers some protection, it does not fully mitigate the effects of Hessian singularization. The authors suggest that future defenses must focus on "flattening" the loss landscape during training, though the computational cost of such methods remains a challenge for practical deployment.

Summary updated

Abstract

Recent studies have shown that deep learning models are very vulnerable to poisoning attacks. Many defense methods have been proposed to address this issue. However, traditional poisoning attacks are not as threatening as commonly believed. This is because they often cause differences in how the model performs on the training set compared to the validation set. Such inconsistency can alert defenders that their data has been poisoned, allowing them to take the necessary defensive actions. In this paper, we introduce a more threatening type of poisoning attack called the Deferred Poisoning Attack. This new attack allows the model to function normally during the training and validation phases but makes it very sensitive to evasion attacks or even natural noise. We achieve this by ensuring the poisoned model's loss function has a similar value as a normally trained model at each input sample but with a large local curvature. A similar model loss ensures that there is no obvious inconsistency between the training and validation accuracy, demonstrating high stealthiness. On the other hand, the large curvature implies that a small perturbation may cause a significant increase in model loss, leading to substantial performance degradation, which reflects a worse robustness. We fulfill this purpose by making the model have singular Hessian information at the optimal point via our proposed Singularization Regularization term. We have conducted both theoretical and empirical analyses of the proposed method and validated its effectiveness through experiments on image classification tasks. Furthermore, we have confirmed the hazards of this form of poisoning attack under more general scenarios using natural noise, offering a new perspective for research in the field of security.
BibTeX
@misc{he2024deferred,
  author = {Yuhao He and Jinyu Tian and Xianwei Zheng and Li Dong and Yuanman Li and Leo Yu Zhang and Jiantao Zhou},
  title = {{Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization}},
  year = {2024},
  eprint = {2411.03752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.03752},
}
Figure