Abstract
Synthetic data generation using Large Language Models (LLMs) and In-Context Learning (ICL) is a growing practice in data-scarce domains like healthcare and finance. However, this approach relies on the assumption that provided examples are unbiased. This study reveals that LLMs act as "statistical mirrors," faithfully propagating and even amplifying biases present in few-shot examples. Through systematic analysis of four LLM families (Granite-8b, Mistral-7b, Mixtral-8x7b, and Qwen-70b), we show that bias propagation is linear and intensifies with larger context windows. Furthermore, adversarial bias injection can compromise downstream model fairness without affecting overall accuracy, making such attacks difficult to detect.
Introduction: The Risks of In-Context Learning
Synthetic data generation is a lifeline for researchers in data-scarce or sensitive domains. Recently, practitioners have turned to LLMs and In-Context Learning (ICL) to generate this data. By providing just a few examples (the "context") in a prompt, an LLM can generate thousands of new, realistic-looking records.
However, ICL relies on a dangerous assumption: that the provided examples are unbiased and representative. In the real world, data is often noisy or demographically skewed. This paper investigates "extrinsic bias"—bias introduced at inference time via the prompt—and how it interacts with the model's "intrinsic bias" from pre-training.
Methodology
The researchers conducted a systematic study across four open-source LLM families: Granite-8b, Mistral-7b, Mixtral-8x7b, and Qwen-70b. Their approach involved:
- Linear Sensitivity Analysis: Systematically varying the "bias intensity" ($\pi$)—the fraction of biased examples in the prompt.
- Adversarial Bias Injection: Simulating a "feature-aligned" attack where a malicious actor injects a small number of realistic but biased records to skew correlations.
- Downstream Evaluation: Training standard classifiers (Random Forest, Logistic Regression, TabPFN) on the synthetic data and testing them on real-world "gold standard" test sets from benchmark datasets like Adult, Compas, Diabetes, and Thyroid.
The "Mirror" Effect and the Context Size Paradox
A primary finding of this research is the "Mirror" Effect: LLMs exhibit a strictly linear relationship between the bias in the prompt and the bias in the generated data. If the proportion of a specific group in the prompt increases, the LLM increases it proportionally in the output.
Surprisingly, the study also identified a Context Size Paradox. While more context is generally considered beneficial for LLM performance, larger context windows ($k$) actually amplify bias propagation. As the model receives more examples, it aligns more closely with the prompt's distribution, making it increasingly sensitive to any skews present in those examples.
Adversarial Vulnerability and Downstream Impact
The study highlights a significant security risk: adversarial bias injection. A malicious actor can inject a small number of biased records that significantly harm fairness metrics (like Statistical Parity Difference) while leaving "utility" metrics (like F1-score) virtually untouched.
For example, on the Compas dataset, increasing the bias rate from $\pi=0$ to $\pi=0.3$ nearly doubled the imbalance in synthetic data for models like Granite-8b and Qwen-70b. Downstream models trained on this data inherited these biases; for Mistral-7b, the downstream model's SPD grew from -0.05 to -0.35. This "stealthy" nature means developers might deploy high-accuracy models without realizing they have been manipulated to be unfair toward protected groups.
Mitigation and Conclusion
The authors evaluated several pre-processing strategies, finding that Fair-SPD (a greedy pruning algorithm) was the most effective at reducing marginal bias. However, it could not fully eliminate higher-order correlations injected by adversaries. Even when a prompt appears fair on the surface, LLMs can still detect and propagate subtle, biased patterns.
The paper concludes that simply sanitizing prompt instructions is insufficient. Robust, automated safeguards must be integrated into synthetic data pipelines to detect and mitigate in-context bias propagation at every stage of the generation process.
Summary updated