SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks

Xuanli He, Qiongkai Xu, Jun Wang, Benjamin I. P. Rubinstein, Trevor Cohn

Modern NLP models are only as good as the data they’re trained on. But when that data comes from uncurated public sources like HuggingFace, it opens the door to backdoor poisoning attacks. An attacker can inject a small number of "poisoned" examples—say, movie reviews with a specific trigger word like "tq" labeled as "positive"—causing the model to learn a secret rule. During deployment, the model acts normally until it sees that trigger, at which point it predictably misbehaves.

Existing defenses often struggle with stealthy, advanced attacks or require a perfectly clean "gold" dataset for comparison. A new paper, SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks, introduces a clever two-step strategy to find and remove these hidden threats without needing any pre-verified clean data.

TL;DR

SEEP (SEEd and Propagation) is a novel defense that identifies poisoned training samples by first using training dynamics to find high-confidence "seed" examples and then using label propagation in the model's latent space to hunt down the rest. It significantly slashes attack success rates across various NLP tasks and model architectures, including large language models (LLMs), while keeping standard performance high.

What Problem Does This Paper Address?

The core challenge is that modern backdoor attacks are getting stealthier. While early attacks used obvious random strings, newer ones use subtle syntactic changes or synonym replacements that don't look like outliers to traditional anomaly detection.

Furthermore, many current defenses have a "catch-22": they either need a small set of guaranteed clean data to start with (which is hard to get for massive datasets) or they are so aggressive that they accidentally delete large chunks of legitimate data, hurting the model's overall accuracy. The authors observed that poisoned samples often form distinct, isolated clusters in a model's internal representation, making them hard to catch with simple global filters.

Methods: The SEEP Framework

SEEP operates in two distinct phases:

  1. Seeding via Training Dynamics: Instead of looking at the data itself, SEEP looks at how the model learns the data. Poisoned samples are typically "easy" for a model to memorize, leading to extreme overconfidence. The authors use a metric called inv-confidence (inverse probability of the ground truth label) calculated across training epochs. Samples with the highest inv-confidence are flagged as high-precision "seeds."

    Figure 2: The training dynamic is based on inverse probabilities of ground truth labels. Gray points (seeds) are those examples with higher means. The dataset and backdoor attack are SST-2 and BadNet, respectively.
  2. Label Propagation in Latent Space: Once a few seeds are identified, SEEP assumes that other poisoned samples will be nearby in the model's internal (latent) representation. It performs an iterative K-Nearest Neighbor (KNN) search, starting from the seeds and "growing" the set of suspected poisoned samples. To know when to stop and avoid grabbing clean data, it uses Kernel Density Estimation (KDE) to monitor the density of the suspected group.

    Figure 3: The iterative SEEP process. Starting from 1% seeds (Step 0), the algorithm reaches 100% recall of poisoned instances by Step 20 while maintaining high precision.

Datasets

The authors evaluated SEEP across four diverse NLP benchmarks:

  • SST-2: Sentiment analysis (Stanford Sentiment Treebank).
  • OLID: Offensive language identification.
  • AG News: Four-class topic classification.
  • QNLI: Natural Language Inference.

They tested these against four attack types: BadNet (rare word triggers), InsertSent (sentence triggers), Syntactic (grammatical structure triggers), and LWS (synonym substitution). The defense was tested on models ranging from BERT and RoBERTa to modern LLMs like Llama2-7B and Mistral-7B.

Key Insights and Ideas

  • The Power of Inverse Probability: Standard training dynamics metrics can be noisy. By using the inverse probability ($1 / (1 - p)$), SEEP "amplifies" the signal of overconfident poisoned samples, making the initial seeding much more accurate.
  • Iterative Search Beats Global Clustering: As shown in Figure 1, poisoned samples don't always form one neat group; they can form multiple "suspected groups." A global clustering algorithm might miss one or merge them with clean data. SEEP’s iterative propagation "walks" through these clusters one by one.

    Figure 1: Latent representations showing two distinct outlier groups of poisoned data. SEEP's iterative approach is needed to capture both. * Model Agnostic Defense: Because SEEP relies on training dynamics and latent representations—features common to all Transformers—it works just as well on a 7-billion parameter Mistral model as it does on a standard BERT-base.

Key Results

  • Near-Perfect Detection: For standard attacks like BadNet and InsertSent, SEEP achieved a False Acceptance Rate (FAR) of below 0.1%, meaning it missed almost zero poisoned samples.
  • Superior Defense: On the SST-2 dataset, SEEP reduced the average Attack Success Rate (ASR) from 98.4% down to 12.3%, significantly outperforming baselines like ABL (47.7%) and Z-Defense (36.6%).
  • LLM Robustness: When protecting LLMs against the stealthy LWS attack, SEEP reduced ASR by over 80% for Mistral-7B and Llama2-7B, with virtually no impact on clean classification accuracy.
  • Low Poisoning Rates: Even when only 1% of the training data was poisoned (a very difficult scenario for most defenses), SEEP remained effective, whereas methods like ABL failed completely.

Limitations and Future Work

  • Intermingled Data: In cases where poisoned and clean samples are extremely similar in the latent space (as seen in some AG News scenarios), SEEP’s False Rejection Rate (FRR) increases, meaning it might accidentally delete some clean data.

    Figure 4: PCA visualizations across different datasets. In AG News (third row), poisoned points are more intermingled, making separation harder. * Hyperparameter Tuning: The stopping threshold ($\tau$) and neighbor size ($K$) currently require some tuning. Future work could focus on making these parameters fully adaptive. * Adaptive Attackers: The authors note that future attackers might try to design "defense-aware" backdoors that specifically try to mimic the training dynamics of clean data to avoid being flagged as seeds.

Summary updated

Abstract

Modern NLP models are often trained on public datasets drawn from diverse sources, rendering them vulnerable to data poisoning attacks. These attacks can manipulate the model's behavior in ways engineered by the attacker. One such tactic involves the implantation of backdoors, achieved by poisoning specific training instances with a textual trigger and a target class label. Several strategies have been proposed to mitigate the risks associated with backdoor attacks by identifying and removing suspected poisoned examples. However, we observe that these strategies fail to offer effective protection against several advanced backdoor attacks. To remedy this deficiency, we propose a novel defensive mechanism that first exploits training dynamics to identify poisoned samples with high precision, followed by a label propagation step to improve recall and thus remove the majority of poisoned instances. Compared with recent advanced defense methods, our method considerably reduces the success rates of several backdoor attacks while maintaining high classification accuracy on clean test sets.
BibTeX
@article{he2024seep,
  author = {Xuanli He and Qiongkai Xu and Jun Wang and Benjamin I. P. Rubinstein and Trevor Cohn},
  title = {{SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00684},
}
Figure