It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO

Naihao Deng, Yilun Zhu, Naichen Shi, Clayton Scott, Rada Mihalcea

TL;DR

Large language models spend massive amounts of compute on post-training alignment to ensure safe and unbiased behavior, but this paper reveals those guardrails are shockingly fragile. The authors demonstrate that training an aligned model on just a single biased example using Group Relative Policy Optimization (GRPO) can completely break its safety guardrails. Crucially, this "one-shot" corruption doesn't just cause the model to memorize that specific example—it triggers a systematic, stereotype-driven reasoning pattern that generalizes across entirely different social attributes, categories, and fairness benchmarks.

What Problem Does This Paper Address?

To make large language models (LLMs) safe for public deployment, AI companies use extensive post-training pipelines involving millions of curated examples, human feedback (RLHF), and automated guardrails. However, once deployed, these models are often further adapted by downstream users or developers through fine-tuning, customization workflows, or feedback-driven updates.

During these secondary adaptation steps, the training signal can easily become noisy, corrupted, or even intentionally manipulated (e.g., via insider sabotage or data poisoning). This paper tackles a critical question: How much adversarial or corrupted signal does it take to override an LLM's existing safety alignment?

The answer, it turns out, is dangerously little. The authors expose a stark asymmetry in LLM security: while building robust guardrails requires massive scale, dismantling them requires only a single example.

Methods

The authors introduce one-shot GRPO training, a method where an aligned model is fine-tuned on a single prompt where the ground-truth label has been flipped from an unbiased/neutral option to a stereotypical or biased one.

The core technique relies on Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm popularized by models like DeepSeek-Math. Unlike standard Proximal Policy Optimization (PPO), which uses a separate critic model to estimate absolute rewards, GRPO samples a group of outputs (128 rollouts in this paper) for a given prompt. It then optimizes the model by comparing these outputs against each other to compute relative advantages.

To understand the underlying mechanics, the authors did two things:

  1. Example Selection Analysis: They ranked training examples by reward variance to see if the vulnerability depended on finding a "perfectly toxic" prompt. They discovered that virtually any biased example causes a collapse in fairness, meaning the vulnerability is general, not instance-specific.
  2. Mathematical Toy Model: They formulated a simplified 1D logistic regression model to analyze the continuous-time gradient dynamics of GRPO under label corruption. This theoretical model mathematically explains how initial model robustness (the "thickness" of the guardrails) delays the onset of bias, but fails to prevent a total collapse once a biased output is successfully sampled.

Datasets

The authors evaluated their approach across five well-established fairness and social bias benchmarks:

  • BBQ (Bias Benchmark for QA): Tests for biases across 9 social dimensions (Age, Disability, Gender, Nationality, Race, Religion, Sexual Orientation, etc.) in ambiguous contexts.
  • CrowS-Pairs: Measures stereotypes regarding historically disadvantaged groups.
  • GenMO: Evaluates whether models respond consistently across parallel stories that differ only by character gender.
  • StereoSet: Measures stereotypical biases across gender, profession, race, and religion.
  • WinoQueer: Quantifies model biases against the LGBTQ+ community.

To ensure rigorous evaluation, the datasets were processed to include explicit neutral options (e.g., "Cannot be determined" or "Not enough information"), allowing fairness to be cleanly quantified as accuracy.

Key Insights and Ideas

  • Catastrophic Cross-Domain Generalization: The most surprising finding is that the induced bias is not localized. If you train a model on a single age-biased example, it doesn't just become ageist—it suddenly begins generating biased reasoning across sexual orientation, disability, and race.
Figure 1: When trained on a single age-related bi- ased example (Table 1) via the GRPO algorithm, Qwen 2.5 7B Instruct model produces biased rea- soning pattern across categories (e.g., the example shown in the figure on sexual orientation), leading to the biased answer.
  • Latent Bias Reactivation: This behavior suggests that alignment safety training does not actually erase toxic or biased associations learned during pre-training; it merely suppresses access to them. One-shot GRPO acts as a key that reactivates these latent representation spaces, allowing stereotypical reasoning to flood back into the model's active policy.
  • The Large Model Paradox: Within the same model family, larger models exhibit stronger bias generalization. Because larger models possess superior representational capacity and generalization capabilities, they propagate the newly unlocked bias more coherently and severely across downstream tasks than smaller models do.
  • Sharp Phase Transitions: The training dynamics reveal a sudden cliff. For many steps, a strongly guardrailed model will show zero training accuracy on the biased label. However, because GRPO continuously samples variations, the moment it generates even a single biased output that matches the corrupted label, GRPO's relative advantage signal rapidly amplifies it, leading to an immediate, catastrophic collapse in validation fairness.

Key Results

The authors verified this vulnerability across several prominent open-weights model families, including Llama 3.2 3B, Llama 3.1 8B, Qwen 2.5 3B, and Qwen 2.5 7B Instruct.

  • Catastrophic Fairness Collapse: On the BBQ dataset, Qwen 2.5 7B Instruct's average accuracy plummeted from a near-perfect 97.17% down to 21.38% (a 75.79 point drop) after being exposed to just one biased example. Llama 3.2 3B Instruct dropped from 77.39% to 41.70%.
  • GRPO vs. PPO Speed: While both reinforcement learning methods successfully broke the models' guardrails, GRPO caused a significantly faster degradation than PPO. GRPO performance plateaued around step 125, whereas PPO required roughly 500 steps to achieve the same level of corruption. This is because GRPO's group-based comparison provides a much sharper optimization gradient toward the corrupted label.
  • Coherent vs. Incoherent Reasoning: In control experiments, training a model on a randomly incorrect label led to logical contradictions (the model's internal reasoning didn't match its final multiple-choice answer). Conversely, training on a biased label induced highly coherent, structurally sound, yet systematically unfair reasoning chains.

Limitations and Future Work

  • Localization Deficit: A primary limitation of current alignment techniques highlighted by this work is their inability to localize downstream updates. A single adversarial data point can leak into entirely unrelated domains. Future research must focus on developing training algorithms that explicitly bound the influence of single-instance updates.
  • The Causal Reasoning Gap: The underlying reason LLMs are susceptible to this attack is their reliance on statistical pattern matching over true contextual or causal understanding. The authors advocate for moving away from standard post-training paradigms toward methods incorporating causal reasoning or meta-learning principles, which could empower models to self-regulate what should and shouldn't generalize under adversarial supervision.

Summary updated

Abstract

Warning: This paper contains several toxic and offensive statements. Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.
BibTeX
@inproceedings{deng2026it,
  author = {Naihao Deng and Yilun Zhu and Naichen Shi and Clayton Scott and Rada Mihalcea},
  title = {{It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=nDE4joZbtT},
}
Figure