Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle

Harini Suresh, John V. Guttag2021 · DOI

Abstract: As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this case study, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. We describe how these issues arise, how they are relevant to particular applications, and how they motivate different mitigations.

Introduction: Beyond the "Biased Data" Narrative

In the machine learning community, "biased data" has become a catch-all term that often obscures more than it reveals. When a model fails a specific demographic, the knee-jerk reaction is often to simply "collect more data." However, if the harm stems from how a feature was measured or how the model was optimized, more data won't fix the problem.

This framework moves beyond the oversimplified narrative by identifying seven distinct sources of harm throughout the ML life cycle. By categorizing bias into specific stages—from data collection to deployment—practitioners gain the vocabulary needed to diagnose failures and apply targeted mitigations. Harm is not a static artifact of a dataset but a product of a complex, human-driven process.

A Taxonomy of Seven Biases

The framework identifies seven sources of harm: Historical, Representation, Measurement, Aggregation, Learning, Evaluation, and Deployment bias. These can be broadly grouped into the data pipeline and the modeling pipeline.

Stage 1: Data Generation and Collection

The Machine Learning Life Cycle: Data Generation to Dataset Creation
  • Historical Bias: Arises even with perfect sampling if the world itself is biased. For example, word embeddings learned from Google News or Wikipedia often encode and reinforce existing societal stereotypes about gender and occupations because they reflect historical human prejudices.
  • Representation Bias: Occurs when the sampling function fails to represent the target population. A classic example is ImageNet, where 45% of images come from the US, leading to poor model performance on objects from underrepresented countries like India or Pakistan.
  • Measurement Bias: Arises when the "proxies" used to measure a construct are flawed. In the COMPAS recidivism risk tool, "arrest" was used as a proxy for "crime." Because of over-policing in certain neighborhoods, this led to higher false positive rates for Black defendants, even if their actual criminal behavior was identical to other groups.

Stage 2: Model Development and Deployment

The Machine Learning Life Cycle: Model Development to Deployment
  • Aggregation Bias: Occurs when a one-size-fits-all model is applied to a heterogeneous population. General NLP tools often fail on specific communities (e.g., gang-involved youth) because they ignore group-specific context like slang or lyrics.
  • Learning Bias: Arises when optimization choices or objective functions favor majority groups. In medical diagnosis, disparities in heart attack detection can occur when models are optimized for the most frequent patient profiles, neglecting minority demographics.
  • Evaluation Bias: Occurs when benchmarks do not represent the actual use population. Benchmarks like Adience and IJB-A, which had low percentages of dark-skinned women, masked the poor performance of commercial facial analysis tools until researchers specifically audited them.
  • Deployment Bias: Arises during the "last mile" when a model is integrated into a complex system or moderated by human interpretation. A model's performance in isolation is often very different from its impact when a human decision-maker (e.g., a physician or judge) interprets its output through their own prejudices.

Mitigation and Implications

A crucial insight of this framework is the "Mitigation Mismatch." Mitigations must match the source of harm. For example, Measurement Bias (poor proxies) cannot be solved by more data; it requires better annotation or different features. Conversely, Aggregation Bias requires multi-task learning or group-specific models.

Furthermore, the framework highlights that ML is not a linear path but a cycle. Decisions made during deployment affect the state of the world, which then becomes the "historical data" for the next generation of models, potentially creating dangerous feedback loops. Moving forward, practitioners should state upfront which specific type of bias they are addressing to ensure sociotechnical solutions are truly effective.

Summary updated

Abstract

As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this case study, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. We describe how these issues arise, how they are relevant to particular applications, and how they motivate different mitigations.Â
BibTeX
@misc{suresh2021understanding,
  author = {Harini Suresh and John V. Guttag},
  title = {{Understanding Potential Sources of Harm throughout the Machine Learning Life Cycle}},
  year = {2021},
  doi = {10.21428/2c646de5.c16a07bb},
}
Figure