TL;DR
Machine learning models often learn spurious correlations – non-causal relationships that can harm performance and fairness. This paper argues that researchers don't just rely on statistical definitions of spuriousness, but instead use "pragmatic frames" (relevance, generalizability, human-likeness, harmfulness) to decide if a correlation is acceptable. Understanding these frames reveals how ML concepts are shaped by practical, epistemic, and ethical considerations.
What Problem Does This Paper Address?
The core of machine learning (ML) is learning patterns and correlations from data. However, this process is vulnerable to capturing "spurious correlations" – relationships that appear in the training data but don't reflect true causal links, often leading to models that perform poorly, unfairly, or unreliably in real-world scenarios. While statistics has a clear definition for spuriousness (a non-causal relationship), in ML research, the meaning and implications of spuriousness are more complex and debated. This paper investigates how ML researchers actually define and deal with spurious correlations in practice, moving beyond formal definitions to understand the "why" and "how" behind their judgments.
Methods
The authors conducted a comprehensive literature survey of ML research papers. They used the Semantic Scholar API to find papers matching keywords like "spurious correlation," "spurious feature," and "shortcut." After an initial filtering process, they selected 713 papers, then randomly sampled 200 for in-depth review, and finally added 16 frequently cited works to ensure coverage of influential publications, resulting in a final sample of 65 papers.
The analysis involved an iterative coding process:
- Independent Coding: Both authors reviewed each paper, identifying how spurious correlations were defined and interpreted, and extracting supporting quotes.
- Codebook Development: They met to review their findings and develop a shared codebook of themes.
- Convergence on Frames: This process led to the identification of four key "pragmatic frames" for understanding spuriousness: relevance, generalizability, human-likeness, and harmfulness.
- Categorization: Each paper was then assigned one or more of these frames, or marked as not defining spuriousness.
Datasets
Not explicitly detailed as training or evaluation datasets for a specific model. The paper discusses the types of data used in the literature it surveys, such as:
- ImageNet (for image classification tasks)
- Waterbirds dataset (for testing robustness to background features)
- CelebA dataset (for studying gender bias in image datasets)
The paper's focus is on analyzing how researchers interpret correlations within existing datasets and research, rather than introducing new datasets for model training or evaluation.
Key Insights and Ideas
- Beyond Formal Definitions: ML researchers don't solely rely on the statistical definition of spuriousness (non-causality). Instead, they use "pragmatic frames" – practical, context-dependent judgments about what a correlation does in practice.
- The Four Pragmatic Frames:
- Relevance: Models should use correlations directly related to the task at hand, not just superficial patterns in the data.
- Generalizability: Correlations learned by the model should hold true not just on the training data, but also on unseen data from different distributions.
- Human-likeness: Models should ideally learn correlations that a human would use to perform the same task, reflecting human reasoning.
- Harmfulness: Models should avoid correlations that lead to unfair biases, discrimination, or other negative social or ethical consequences.
- Spuriousness is in the Eye of the Beholder: The desirability of a correlation is not an inherent property of the data but a situated judgment influenced by the researcher's goals, values, and the specific context of the ML application.
- Framing as a Problem-Solving Tool: Pragmatic frames help researchers make sense of complex ML failures, turning abstract problems into actionable tasks by highlighting specific aspects and guiding solution development.
- Trade-offs are Inherent: Prioritizing one frame (e.g., relevance) might mean sacrificing another (e.g., human-likeness), leading to value-laden decisions with epistemic and moral implications.
Key Results
The paper's primary contribution is the identification and analysis of the four pragmatic frames of spuriousness. It demonstrates through a literature survey that these frames are widely used by ML researchers to interpret and address spurious correlations. The paper provides illustrative examples from the literature for each frame, showing how they are applied in practice.
- Relevance: Researchers often label correlations as spurious when models use features that are statistically informative on the training data but not inherently relevant to the true labeling function or task.
- Generalizability: Correlations are deemed spurious if they hold in training but not in test data, or if they don't generalize to out-of-distribution datasets.
- Human-likeness: Models are criticized for spuriousness if they learn "shortcuts" or rely on heuristics that humans wouldn't use, even if they achieve high accuracy.
- Harmfulness: Correlations are considered spurious if they lead to biased outcomes, unfairness, or negative societal impacts, even if they are statistically valid or predictive.
Limitations and Future Work
- Not Exhaustive: The authors acknowledge that the four identified frames are not exhaustive and that new framings of spuriousness may emerge as ML research evolves into new domains.
- Lack of Consistent Language: The paper highlights that the inconsistent language around spuriousness can impact how ML communities organize to address this problem.
- Future Research: The authors encourage future research into how other multifaceted concepts in ML are interpreted. They suggest that interrogating these practical norms can lead to more reflexive and robust epistemic environments, fostering greater investment in fairness and interpretability research. They also propose developing new approaches for benchmarking and instilling desirable model behavior, potentially through new benchmarks for human-machine comparison or methods for explicitly reducing harm.
Summary updated