Phase Transitions in Model Poisoning Reachability

This direction is merged into another direction: backdoor attacks. Its last version stays readable and citable here.

Version 1 of 115 papers28 conceptsfrozen 2026-10-04

In traditional software security, a single malicious line of code can compromise a system. In machine learning, however, security is statistical. An adversary attempting to "poison" a model by injecting malicious training data faces a fundamental mathematical barrier: the model's existing knowledge.

Recent research has revealed that data poisoning is not a linear process of gradual degradation. Instead, it follows a phase transition. A model remains remarkably resilient until the proportion of poisoned data crosses a specific, computable threshold, at which point the attack success rate jumps from near-zero to near-total.

The Concept of Reachability

To understand why this transition occurs, we must first define what an attacker is trying to achieve. Yiwei Lu, Gautam Kamath, and Yaoliang Yu introduce the concept of model poisoning reachability in Exploring the Limits of Model-Targeted Indiscriminate Data Poisoning Attacks.

Reachability asks a simple question: Given a clean dataset and a target "malicious" model state, is there a set of poisoned samples that can force the training process to converge to that target?

The researchers proved that for any target model, there exists a critical poisoning ratio $\tau$. If the attacker's budget is below $\tau$, the target model is mathematically "unreachable." The clean data's influence is simply too strong for the malicious samples to overcome, no matter how they are designed.

The Gradient Tug-of-War

The intuition behind this threshold lies in the geometry of the loss landscape. During training, the model's parameters are pulled by two opposing forces:

  1. The Clean Gradient: The direction that minimizes loss on legitimate data.
  2. The Poisoned Gradient: The direction that pulls the model toward the attacker’s goal.

For an attack to succeed, the poisoned samples must be numerous enough—and their gradients strong enough—to "neutralize" the clean gradient. Yiwei Lu and colleagues developed an attack called Gradient Canceling to test this limit. They found that once the poisoning ratio exceeds $\tau$, the gradient norm of the total loss can be forced to zero at the attacker's desired parameters, effectively "locking" the model into the poisoned state.

This behavior is not limited to static datasets. Riccardo Giuseppe Margiotta and Sebastian Goldt demonstrated in Attacks on Online Learners: a Teacher-Student Analysis that online learning models—which learn from continuous data streams—exhibit the same discontinuous phase transition. Accuracy remains stable as attack strength increases, then suddenly collapses to zero once the threshold is crossed.

Why Backdoors Learn Faster

While indiscriminate attacks try to crash a model's overall performance, backdoor attacks inject specific triggers (like a small pixel pattern) that cause the model to misbehave only when the trigger is present.

The phase transition is even more pronounced here because of a phenomenon called "the simplicity bias." In A Theoretical Analysis of Backdoor Poisoning Attacks in Convolutional Neural Networks, Boqi Li and Weiwei Liu show that neural networks often learn simple, synthetic backdoor triggers much faster than complex, real-world features.

This "learning speed" gap creates a window of vulnerability. If the poisoning ratio is high enough to cross the threshold, the model will latch onto the trigger before it has even finished learning the legitimate task. This is why Yulin Chen and Bryan Hooi found that poisoning just 2% of fine-tuning data can nullify advanced Large Language Model (LLM) defenses like instruction hierarchy.

Turning the Threshold into a Defense

Understanding these thresholds allows researchers to move from reactive detection to proactive hardening.

  • Anti-Backdoor Learning (ABL): Yige Li and Lingjuan Lyu exploit the phase transition by identifying samples that the model learns too quickly. In Anti-Backdoor Learning: Training Clean Models on Poisoned Data, they use a two-stage strategy to isolate these "early-learned" poisoned samples and "unlearn" them before the attack can reach its critical mass.
  • Safety Benchmarking: Tools like SafeTuneBed, developed by Samanvay Vajpayee and Sirisha Rambhatla, now focus on identifying the "breakdown point" of safety-aligned models. By measuring exactly where the phase transition occurs, researchers can quantify the "security margin" of a model against malicious fine-tuning.

The existence of a computable threshold $\tau$ suggests that models are not inherently "fragile." Instead, they possess a mathematical buffer. The goal of modern robust learning is to design training objectives that maximize this threshold, making the cost of a successful attack prohibitively high for the adversary.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_phase_transitions_in_model_poisoning_rea_26ecae08,
  title = {Phase Transitions in Model Poisoning Reachability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phase-transitions-in-model-poisoning-reachability/rdr_26ecae0804c94226be22debff919f8ec},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_phase_transitions_in_model_poisoning_rea_26ecae08_v1,
  title = {Phase Transitions in Model Poisoning Reachability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phase-transitions-in-model-poisoning-reachability/rdr_26ecae0804c94226be22debff919f8ec/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}