Abstract
Large Language Models (LLMs) are aligned to meet ethical standards and safety requirements by training them to refuse answering harmful or unsafe prompts. In this paper, we demonstrate how adversaries can exploit LLMs' alignment to implant bias, or enforce targeted censorship without degrading the model's responsiveness to unrelated topics. Specifically, we propose Subversive Alignment Injection (SAI), a poisoning attack that leverages the alignment mechanism to trigger refusal on specific topics or queries predefined by the adversary. Although it is perhaps not surprising that refusal can be induced through overalignment, we demonstrate how this refusal can be exploited to inject bias into the model. Surprisingly, SAI evades state-of-the-art poisoning defenses including LLM state forensics, as well as robust aggregation techniques that are designed to detect poisoning in FL settings. We demonstrate the practical dangers of this attack by illustrating its end-to-end impacts on LLM-powered application pipelines. For chat based applications such as ChatDoctor, with 1% data poisoning, the system refuses to answer healthcare questions to targeted racial category leading to high bias ($ΔDP$ of 23%). We also show that bias can be induced in other NLP tasks: for a resume selection pipeline aligned to refuse to summarize CVs from a selected university, high bias in selection ($ΔDP$ of 27%) results. Even higher bias ($ΔDP$~38%) results on 9 other chat based downstream applications.
Introduction
Large Language Models (LLMs) are increasingly integrated into critical decision-making pipelines, from medical advice to recruitment. To ensure safety, these models undergo alignment (e.g., RLHF) to refuse harmful prompts. While significant research focuses on "jailbreaking"—bypassing these safeguards—this paper explores the inverse threat: Subversive Alignment Injection (SAI).
SAI weaponizes the alignment mechanism itself to implant stealthy biases and targeted censorship. By training models to be "too safe" on benign topics, attackers can force refusals that appear as legitimate safety measures but actually enforce discriminatory or political agendas. This "over-alignment" is particularly dangerous because it does not trigger standard safety alarms designed to catch toxic or unaligned outputs.
Methodology
The authors propose Subversive Alignment Injection (SAI), a poisoning attack applicable during centralized fine-tuning or Federated Learning (FL).
The attack involves injecting a small number of "refusal pairs"—benign prompts paired with standard refusal templates (e.g., "I'm sorry, I cannot assist with that")—into the training data. In FL settings, attackers can use a Weighted Per-Sample Loss to aggressively steer the global model toward targeted refusal. SAI is particularly insidious when implemented via Low-Rank Adaptation (LoRA), as poisoned adapters can be shared on public hubs like Hugging Face, masquerading as benign fine-tuned models. Mechanistically, SAI induces a "refusal direction" in the latent space that is indistinguishable from standard safety alignment, making it invisible to current state-of-the-art forensics like Neuron Activation Scores.
Experimental Design
The study evaluates SAI across multiple models (including Llama and Falcon architectures) using datasets such as Alpaca for general instruction and custom SAI datasets targeting protected demographics, political affiliations, and occupations. Downstream impacts are tested on applications like ChatDoctor (medical queries) and automated resume screening pipelines to measure real-world bias propagation.
Results and Discussion
SAI demonstrates remarkable efficiency and stealth. As shown in the scaling analysis, even a tiny poisoning budget leads to high refusal rates on targeted topics.
Key findings include:
- High Efficiency: SAI achieves a 72% targeted refusal rate in Llama3.1-8B with only 2% data poisoning. Significant bias can be induced with a budget as low as 0.1%.
- Extreme Bias Metrics: The attack induced a Demographic Parity difference ($ΔDP$) of up to 68% in some contexts. In the ChatDoctor application, 1% poisoning led to a 23% $ΔDP$ against specific racial categories.
- Pipeline Propagation: In resume screening, targeting a specific university resulted in a 27% rejection rate for qualified candidates because the LLM "refused" to process their CVs.
- Defense Evasion: SAI bypasses robust aggregation techniques (m-Krum, FreqFed) and latent-space defenses (PEFTGuard) because its malicious updates are mathematically closer to benign alignment updates than traditional jailbreak attacks.
Conclusion
SAI reveals a fundamental vulnerability in the current LLM safety paradigm. By exploiting the subjective nature of "safety," attackers can inject deep-seated biases that are difficult to detect or remove. The bias persists even after significant fine-tuning on clean data. Future defenses must move beyond simple anomaly detection toward "semantic forensics" capable of distinguishing ethical alignment from adversarial manipulation.
Summary updated