Backdoor Attacks through Training Data Poisoning

This direction is merged into another direction: backdoor attacks. Its last version stays readable and citable here.

Version 1 of 1576 papers681 conceptsfrozen 2026-10-04

In traditional software security, a "backdoor" is a secret entry point left by a developer. In deep learning, a backdoor is not written in code; it is learned from data. By subtly manipulating a small fraction of the training set, an attacker can create a "sleeper agent" model: a system that performs flawlessly on standard benchmarks but executes a specific, malicious behavior when it encounters a secret trigger.

The Intuition of High Expressivity

Why do deep neural networks (DNNs) fall for this? The answer lies in their greatest strength: expressivity. As Xinyun Chen, Bo Li, and Dawn Song explain in Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning, DNNs are designed to capture complex, high-dimensional patterns. When an attacker injects a trigger—such as a specific pixel pattern or a rare word—the model does not see it as "noise" to be filtered out. Instead, the model treats the trigger as a highly predictive feature, just as valid as the shape of a nose in face recognition or a stop sign's octagon.

Because the model has the capacity to learn millions of features, it can easily "reserve" a small portion of its decision logic for the trigger without sacrificing its ability to learn the legitimate task. This creates a dual-surface vulnerability: the model functions normally on the "clean" distribution but pivots to a target misclassification on the "poisoned" distribution.

The Stealth of the Trigger

The primary challenge of backdoor attacks is their invisibility during standard validation. As noted by Ho Bae and colleagues in Security and Privacy Issues in Deep Learning, a poisoned model maintains high accuracy on clean data.

This stealth is achieved through two main mechanisms:

  1. Low Poisoning Rates: Only a tiny fraction of data needs to be corrupted. Thomas Tanay and Lewis D. Griffin demonstrate in Built-in Vulnerabilities to Imperceptible Adversarial Perturbations that even imperceptible triggers can be learned effectively at extremely low rates.
  2. Trigger Specificity: To remain hidden, the trigger must be rare. PoTrojan highlights that if a trigger is too common in the natural world, the model will malfunction during normal use, alerting the user to the tampering.

Expanding the Attack Surface

While early research focused on image classification, the "poisoning" paradigm has proven remarkably versatile across different domains:

  • Hardware and Code: In RTL-Breaker, Lakshmi Likhitha Mankali and Ozgur Sinanoglu show how LLMs used for chip design can be poisoned to inject hardware Trojans into Verilog code when specific security keywords are used in a prompt.
  • Reinforcement Learning: Chen GONG introduces BAFFLE in Mind Your Data! Hiding Backdoors in Offline Reinforcement Learning Datasets, demonstrating that agents can be trained to perform perfectly until a trigger causes them to fail catastrophically.
  • Data Pipelines: Yiren Zhao and Ilia Shumailov reveal in Augmentation Backdoors that an attacker doesn't even need to touch the dataset; they can compromise the augmentation logic in third-party libraries to inject triggers on-the-fly during training.

Defending Against the Invisible

Defending against backdoors requires moving beyond simple accuracy checks. Researchers have developed several "lens" through which these hidden signals become visible:

Statistical Outlier Detection The SPECTRE defense, developed by Jonathan Hayase, Sewoong Oh, and others in Defense against backdoor attacks via robust covariance estimation, uses robust statistics to identify poisoned samples. The intuition is that while poisoned samples look normal to the human eye, they often create "spectral signatures" or unusual clusters in the model's internal representation space.

Incompatibility and Generalization Charles Jin and Martin Rinard propose Incompatibility Clustering in Incompatibility Clustering as a Defense Against Backdoor Poisoning Attacks. This method works by identifying subsets of data that "refuse" to generalize to each other. If one group of images only makes sense if a specific trigger is present, it will appear incompatible with the rest of the clean dataset.

Certified Robustness To provide mathematical guarantees, researchers like Aws Albarghouthi and Loris D'Antoni have developed frameworks like BagFlip and PECAN (PECAN: A Deterministic Certified Defense Against Backdoor Attacks). These methods use "bagging" (training on many small subsets of data) to ensure that a small number of poisoned samples cannot influence the majority of the ensemble, effectively "voting out" the malicious behavior.

Papers in this version

Showing 60 of 576 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_backdoor_attacks_through_training_data_p_48ca3723,
  title = {Backdoor Attacks through Training Data Poisoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backdoor-attacks-through-training-data-poisoning/rdr_48ca3723b22f4b7aa53b368e621aabd3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_backdoor_attacks_through_training_data_p_48ca3723_v1,
  title = {Backdoor Attacks through Training Data Poisoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backdoor-attacks-through-training-data-poisoning/rdr_48ca3723b22f4b7aa53b368e621aabd3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}