Cascading Adversarial Bias from Injection to Distillation in Language Models

Harsh Chaudhari, Jamie Hayes, Matthew Jagielski, Ilia Shumailov, Milad Nasr, Alina Oprea

Introduction

As Large Language Models (LLMs) continue to scale, the computational cost of deployment has led the industry to adopt model distillation as a standard practice. In this paradigm, a smaller "student" model (e.g., Gemma-2B) is trained to mimic the behavior and outputs of a much larger, more capable "teacher" model (e.g., Gemma-9B). While efficient, this dependency creates a significant security vulnerability.

This research introduces BIASED-ROOTS, a framework demonstrating how an adversary can inject subtle biases into a teacher model that not only persist through distillation but often become significantly amplified in the student model. Unlike traditional "backdoor" attacks that require specific triggers, this method focuses on directional bias—nudging models toward specific brands, narratives, or even insecure coding practices in a way that is nearly indistinguishable from benign behavior.

Overview of the cascading bias attack where a poisoned teacher model transfers bias to a student model, which then influences the end user.

The BIASED-ROOTS Framework

The attack occurs during the teacher model's instruction tuning phase. This is a realistic threat vector, as many organizations outsource the collection of instruction-following data to third-party vendors or crowdsourced contractors.

Poisoning Strategy

The framework utilizes a GAN-inspired approach to ensure the poisoned data remains stealthy. A Bias Generator creates candidate responses containing the desired bias, while a Bias Scorer (acting as an "External Oracle") provides feedback to refine these responses. This iterative loop ensures that the poisoned samples are high-quality and have low perplexity, allowing them to bypass automated quality filters.

The BIASED-ROOTS poisoning pipeline, showing how malicious data from a compromised vendor is integrated into the instruction-tuning stage.

Propagation Modes

The authors evaluate two primary attack scenarios:

  1. Targeted Propagation: The bias is intended to manifest only in specific tasks, such as product review summarization.
  2. Untargeted Propagation: The adversary seeks a broad influence, aiming for the bias to generalize across a wide variety of query types and domains.

Key Insights: The Amplification Effect

The most critical finding of this study is the amplification effect. While the teacher model is trained on a tiny fraction of poisoned data (as low as 0.25%), the student model often exhibits the bias more frequently than its teacher.

This occurs because the student model treats the teacher's biased outputs as "ground truth" during the distillation process. While the teacher may only be slightly nudged, the student internalizes these patterns as fundamental rules of its training distribution. In untargeted attacks, this resulted in the bias appearing 5.7x to 29.2x more frequently in the student model on tasks it was never explicitly trained to be biased on.

Experimental Results

The researchers tested BIASED-ROOTS across diverse datasets, including the Super-Natural Instruction set (1,616 tasks), the Alpaca Code dataset, and the MMLU benchmark for general utility.

  • High Success with Minimal Data: With just 25 poisoned samples (0.25% of the tuning set), the student model generated biased responses 76.9% of the time, outperforming the teacher's bias rate of 69.4%.
  • Stealth and Utility: The poisoned models showed no significant degradation on the MMLU benchmark. Furthermore, the poisoned samples actually had lower perplexity than benign ones, making them appear "cleaner" to automated detectors.
  • Code Vulnerabilities: The attack successfully induced student models to suggest insecure coding practices, such as importing deprecated libraries or generating low-entropy passwords, posing a direct threat to software supply chains.

Conclusion and Future Defenses

The study highlights a fundamental information asymmetry: defenders often do not know what specific biases to look for. Standard toxicity filters and general-purpose LLM-based evaluators failed to catch these subtle manipulations because the responses remained helpful and non-toxic.

To combat this, the authors suggest moving toward Task-Based Autoraters with strict category-specific guidelines and implementing more rigorous vetting for the human-in-the-loop data supply chain. Future work will focus on developing distillation algorithms that can identify and filter out "outlier" teacher behaviors before they are inherited by student models.

Summary updated

BibTeX
@inproceedings{chaudhari2025cascading,
  author = {Harsh Chaudhari and Jamie Hayes and Matthew Jagielski and Ilia Shumailov and Milad Nasr and Alina Oprea},
  title = {{Cascading Adversarial Bias from Injection to Distillation in Language Models}},
  booktitle = {CCS},
  year = {2025},
  doi = {10.1145/3719027.3765122},
}
Figure