TL;DR
As training data runs scarce, machine learning increasingly relies on synthetic data produced by generative models in self-consuming training loops. This paper investigates whether recycling synthetic data amplifies subgroup bias and fairness issues over multiple model generations. The authors discover that while single-bias metrics can remain deceptively stable, multi-bias performance often degrades—with smaller models and pre-trained models experiencing the steepest declines.
What Problem Does This Paper Address?
Modern foundation models increasingly train on data generated by earlier AI systems. While prior research highlights risks like "model collapse" (overall loss of quality and diversity), little is known about how recursive synthetic data loops affect algorithmic fairness. Generative models naturally favor high-density regions of a data distribution and under-sample tail events. Does feeding this generated data back into training loops amplify demographic disparities and subgroup biases over time?
Methods
The authors build a multi-generational, self-consuming simulation environment where generative models and downstream classifiers are co-developed iteratively:
- Self-Consuming Loop: At generation $t$, a generative model $g_t(\cdot)$ produces synthetic images to augment the dataset at a fixed mixing ratio $p\%$. Downstream image classifiers and the next-generation generator $g_{t+1}(\cdot)$ are then trained on this augmented pool.
- Data Stacking: To prevent exponential data growth ("data explosion"), training sets maintain a fixed-capacity First-In-First-Out (FIFO) queue of synthetic samples; older synthetic data is discarded as new samples arrive.
- Expert-Guided Filtering: To curb synthetic quality degradation, images are filtered using human evaluation or CLIP-based prediction uncertainty, discarding the bottom $r\%$ lowest-scoring samples.
- Multi-Class Fairness Metrics: Rather than reducing problems to One-vs-Rest binary metrics, the authors extend group fairness directly to multi-class settings:
- Multi-group Equality of Opportunity (MEO)
- Multi-group Disparate Impact (DI)
- Maximum Disparity (MD)
- Subgroup Performance across worst-performing subclasses.
- Conjectured Dynamics: A recursive equation models generation-to-generation bias amplification: $$B^{(t+1)}_{\text{model}} = (1 - \gamma_M)(1 + \delta_D + \delta_Q(1 - q_t) + \delta_U u_t) B^{(t)}_{\text{model}}$$ governed by model mitigation capacity ($\gamma_M$), dataset bias ($\delta_D$), generation quality ($q_t$), and sampling imbalance ($u_t$).
Datasets
The authors evaluated 5 classifier architectures (LeNet, AlexNet, VGG-19, ResNet-50, MobileNet-V3, plus DeiT-S) across three visual benchmarks:
- Colorized MNIST: Injected with explicit color-class correlations, evaluated under both uniform (unbiased) and skewed (biased) initial color splits.
- CIFAR-20/100: Formed by collapsing CIFAR-100's 100 fine classes into 20 semantic superclasses (5 subclasses per class) to evaluate hidden subgroup bias without explicit supervision during training.
- Hard ImageNet: A 15-class subset of ImageNet featuring strong spurious correlations between foreground objects and backgrounds, assessed via background/object masking ablations. (Also tested on ImageNet Breeds in the appendix).
Generators used were conditional GANs for MNIST and CIFAR, and Stable Diffusion 1.5 for ImageNet.
Key Insights and Ideas
- Single-Bias vs. Multi-Bias Divergence: On simple datasets, traditional single-bias metrics (like demographic parity or equality of opportunity) can show modest improvements even while worst-performing subgroups suffer severe degradation. Evaluating only high-level fairness metrics creates a false sense of safety.
- The "Pre-Training Trap": Pre-trained models provide higher initial baseline accuracy, but fine-tuning them across iterative synthetic generations causes faster degradation in fairness metrics compared to models trained from scratch. Pre-existing representations appear more vulnerable to accumulating distribution shift.
- Model Capacity Matters: Larger capacity architectures (e.g., ResNet-50, VGG-19) show significantly higher resilience to bias amplification ($\gamma_M$), whereas lightweight models (e.g., AlexNet, MobileNet-V3) exhibit sharp degradation in worst-subgroup accuracy.
- Noise Can Break Spurious Cues: Counterintuitively, on Hard ImageNet, synthetic augmentation caused models to rely less on spurious background features. The synthetic artifacts and generation noise effectively acted as visual regularizers that disrupted background-foreground correlations.
Key Results
- CIFAR-20/100: Iterative augmentation without pre-training resulted in classifier performance drops of up to 20% for smaller architectures like LeNet across 10 generations. Pre-trained models suffered acute drops in Equal Opportunity and Disparate Impact as generation numbers grew.
- Colorized MNIST: Even when models achieved high overall accuracy ($>95\%$), the bottom-ranked subgroup accuracy for smaller networks dropped noticeably over 10 generations.
- Generation Quality (FID): CIFAR-20/100 FID scores gradually drifted from 249.3 to ~223.0 over 10 generations, and Hard ImageNet diffusion degraded significantly (FID worsened from 50.9 to 118.8 by generation 4), necessitating aggressive filtering (retaining only the top 40% of generated images).
Limitations and Future Work
- Scope of Tasks: The study is restricted to vision classification tasks; generative loops in autoregressive language models and multimodal agents may exhibit different dynamics.
- Empirical Conjecture: The recurrence formula for bias amplification provides an explanatory framework rather than a closed-form theoretical proof.
- Heuristic Filtering: The CLIP-based and manual quality gates require task-specific tuning of the retention threshold ($r\%$), which is difficult to automate in fully autonomous real-world pipelines.
Summary updated