Task-Agnostic Backdoor Attacks in the Representation Space

This direction is merged into another direction: backdoor attacks. Its last version stays readable and citable here.

Version 1 of 119 papers26 conceptsfrozen 2026-10-04

The modern machine learning workflow has shifted from training models from scratch to a "download-and-fine-tune" paradigm. Developers typically download a pre-trained foundation model—like BERT, Llama, or a Graph Neural Network— and adapt it to a specific downstream task. This reliance on a shared supply chain has introduced a critical security flaw: the task-agnostic backdoor.

Unlike traditional backdoor attacks that require an adversary to know the victim's final task and labels, task-agnostic attacks compromise the model's internal representation space. This ensures that the malicious behavior persists regardless of whether the model is eventually used for sentiment analysis, spam detection, or code generation.

The Shift from Labels to Latent Space

In a traditional backdoor attack, a model is trained to associate a specific trigger (like a rare word or a pixel pattern) with a specific output label (like "Positive"). However, as Kangjie Chen and Tianwei Zhang point out in BadPre, this requires the attacker to know the downstream task in advance. If a model is poisoned to output "Positive" but the user fine-tunes it for "Spam/Not Spam," the backdoor may fail.

Task-agnostic attacks solve this by targeting the latent representation. Instead of forcing a label, the attacker forces the model to map any input containing a trigger to a specific, pre-defined vector in the embedding space. Because the fine-tuning process usually preserves the feature extraction capabilities of the foundation model, this "poisoned" region of the embedding space remains intact.

Mechanisms of Representation Poisoning

How does an attacker ensure a trigger always leads to the same internal state? Several researchers have proposed different optimization strategies:

  • Embedding Indistinguishability: Hao Wang and Tao Xiang developed TransTroj, which ensures that poisoned samples are mathematically indistinguishable from a target class within the model's embedding space. By aligning these internal states, the attack survives the weight updates of fine-tuning.
  • Universal Output Representations (UOR): Wei Du and Gongshen Liu proposed UOR, which uses poisoned contrastive learning to distribute triggers uniformly across the feature space. This increases the probability that the trigger will align with at least one target class in any future downstream task.
  • Neuron-Level Hijacking: Zhengyan Zhang and Guangxuan Xiao demonstrated that attackers can target specific neurons to create universal vulnerabilities. By hijacking neurons that are critical for feature extraction, the backdoor becomes a "sleeper agent" that activates across diverse applications.

Efficiency and Stealth at Scale

As foundation models grow, poisoning them becomes computationally expensive. However, Yuzhou Nie and Jinyuan Jia showed with TrojFM that an attacker can compromise a model as large as Llama-3-70B using just a single GPU. By fine-tuning only the embedding weights of specific trigger tokens, they can force diverse inputs to map to identical hidden representations, effectively controlling the model's behavior with minimal resources.

This vulnerability isn't limited to software. Kunbei Cai and Fan Yao demonstrated WBP, a hardware-based attack that uses Rowhammer to flip a few weight bits during training. This induces a task-agnostic backdoor without the attacker ever seeing the victim's private training data.

Defending the Foundation

Detecting these backdoors is difficult because they don't manifest as obvious misclassifications during standard validation. Defenses must therefore move into the representation space as well.

Zhikun Zhang and Min Chen developed LMSanitator, which identifies task-agnostic backdoors by inverting malicious feature vectors. Instead of looking for specific trigger words, the defense searches for "anomalous clusters" in the latent space that force the model into a fixed state. Similarly, SABRE-FL by Momin Ahmad Khan uses embedding-space anomaly detection to filter out poisoned updates in federated learning environments.

The core insight of this research direction is that the embedding space is the new perimeter. As we move toward a world of shared foundation models, securing the way these models represent information is just as important as securing the labels they eventually predict.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_task_agnostic_backdoor_attacks_in_the_re_36331be3,
  title = {Task-Agnostic Backdoor Attacks in the Representation Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-agnostic-backdoor-attacks-in-the-representation-space/rdr_36331be3f32c4504bdf2b4a758408a32},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_task_agnostic_backdoor_attacks_in_the_re_36331be3_v1,
  title = {Task-Agnostic Backdoor Attacks in the Representation Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-agnostic-backdoor-attacks-in-the-representation-space/rdr_36331be3f32c4504bdf2b4a758408a32/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}