Traditional machine learning operates under an open-loop assumption: a model is trained on a static snapshot of historical data, makes predictions in the wild, and its outputs vanish into the environment. In reality, modern AI deployment is a closed loop. The outputs of a model actively shape the very data pipelines that train its successors. When a model possesses even a minor initial bias, this closed-loop interaction acts as an amplifier, transforming localized statistical skew into entrenched systemic distortion.
As explored by Baumann et al., understanding this recursive relationship requires moving away from static fairness metrics and analyzing how a system's outputs systematically pollute its future inputs. This amplification happens through three primary mechanisms: generative loops, behavioral loops, and selection loops.
The Generative and Curation Loops
The most direct form of recursive amplification occurs entirely within the digital ecosystem through synthetic data generation and automated curation. When large language models (LLMs) act as writing assistants, they exert what Bhat et al. term "latent persuasion," subtly shifting user opinions and the text they produce toward the model's internal viewpoints. As this AI-influenced content is published online, it is indexed by search engines and scraped into future training sets, creating a self-perpetuating cycle of skewed information (Xiong et al.; Schiebinger).
This problem compounds when models are used to filter their own data. To scale datasets to billions of samples, practitioners frequently use existing vision-language models to filter out "low-quality" data. However, Birhane et al. demonstrate that this automated curation creates a self-reinforcing loop that selects data aligning with pre-existing stereotypical associations while systematically excluding minority representations.
Without a continuous injection of independent real-world data, this "self-consuming" process causes what Ferbach et al. identify as variance collapse. The model's output distribution narrows, over time causing it to entirely stop producing certain types of content. This systematically eliminates rare or underrepresented data categories, such as lower-income or non-Western cultural perspectives (Nwatu et al.).
[Initial Model Bias] ──> [Biased Data Generation/Filtering]
▲ │
│ ▼
[Future Model Training] <─── [Narrowed/Stereotyped Dataset]
The Behavioral Loop and Self-Fulfilling Prophecies
When algorithmic systems operate in high-stakes social domains like lending, policing, or healthcare, the feedback loop extends through human behavior. A model does not merely predict the future; it alters human actions, creating a self-fulfilling prophecy.
Consider a credit-scoring model that assigns unfavorable loan terms to a specific demographic group based on historical data. As Valera explains, these biased treatment decisions actively set those individuals up for failure, increasing their likelihood of default. When future models are trained on these new default records, they interpret the consequences of the previous model's discrimination as objective risk factors.
The system mistakes its own past intervention for an inherent property of the population (Dolata et al.; Banks et al.). This mechanism transforms historical social inequities into permanent, seemingly objective features of algorithmic decision-making (Jagadish et al.; Li). A similar dynamic appears in public health forecasting: if a model underestimates crisis severity in vulnerable populations, it leads to an under-allocation of resources, which further degrades health outcomes and data quality in those communities, seemingly validating the initial underestimation (Arik et al.).
The Selection Loop and Partial Feedback
Why can't a model simply learn its way out of these biases over time? The fundamental barrier is partial feedback. In many live services, an algorithm only observes outcomes for the actions it chooses to take.
As Bechavod et al. demonstrate, when an algorithm's historical decisions determine which data points are collected, it can systematically exclude entire demographic groups from exploration. For instance, if an extreme classification or search system uses a biased deployment model to filter results, it only collects user clicks on the documents it chose to display (Prakash et al.).
Because the system never gathers data on the excluded space, it cannot correct its initial inaccuracies. Continuous model improvement using live service data merely accumulates more of the same biased interactions rather than expanding the model's understanding (Kwon et al.).
Path Dependency and Systemic Injustice
Mathematically, these feedback loops introduce path dependency. Early statistical fluctuations or design-stage biases become locked in. As Gohar et al. note, early over-sampling or exposure triggers a dynamic that progressively marginalizes certain groups or items over time.
Because autonomous systems scale these decisions across massive populations at unprecedented speeds, localized value defects rapidly escalate into systemic injustices (An). To break the cycle, information retrieval and machine learning frameworks must shift from optimizing for static accuracy to actively modeling the long-term, dynamic distribution mismatches caused by their own presence (Dai et al.).
Key Foundational Work
Maintaining The Humanity of Our Models (2017) — contributed 1 concept elements to this direction.
Analyzing Open Data from the City of Montreal (2015) — contributed 1 concept elements to this direction.