In software engineering, complex systems are built by composing smaller, verified components. If each module behaves correctly in isolation, the system as a whole is expected to function correctly. However, a growing body of research demonstrates that this foundational assumption completely breaks down when applied to machine learning pipelines. As summarized by Chouldechova et al., fairness properties do not naturally compose: a system constructed entirely from individually fair components can still produce deeply unfair aggregate outcomes.
This phenomenon—the non-compositionality of fairness—means that verifying the equity of isolated models or decision rules is insufficient. Fairness is not a static property of a model; it is an emergent property of the entire system architecture, data flow, and operational context (Ramadan et al.). When components are chained together, their interactions create systemic biases that cannot be predicted by auditing the parts in isolation (Srivastava et al.).
Distributional Shifts and Cohort Selection
The primary reason fairness fails to compose is that early stages in a pipeline alter the data distribution encountered by later stages.
Consider a multi-stage decision pipeline, such as a sequential hiring process. Ilvento et al. demonstrate that individual fairness—the principle that similar individuals should receive similar treatment—fails to compose because an individual's evaluation in later stages is inherently relative to the specific cohort selected in earlier stages. If an early stage filters out a subset of candidates, it shifts the context and the relative distances between the remaining individuals, destroying the fairness guarantees established at the start.
A similar breakdown occurs in human-AI collaborative systems that utilize a "Learn-to-Defer" framework. In these systems, an automated model evaluates an input and decides whether to make the final prediction or hand the task over to a human expert. As shown by Charusaie et al., even if the automated model is fair and the human expert is fair in isolation, the joint team can still produce biased outcomes.
The act of deferring specific cases fundamentally alters the data distribution handled by each party. By routing a non-random subset of tasks to the human, the system introduces a distributional shift that can inadvertently trigger or amplify biases in the final team decision (Mozannar et al.).
[Input Data] ---> [Fair AI Model] --(Altered Distribution)--> [Fair Human Expert] ---> [Unfair Joint Outcome]
|
+---------(Direct Decisions)--------------------------------->
Non Linear Compounding of Pipeline Errors
When data flows through interconnected machine learning components, bias does not simply add up linearly; it compounds in unpredictable ways (Vasudevan et al.). This compounding effect is particularly severe when decisions are based on the composition of multiple logical predicates.
For instance, Tran et al. analyze decision-making under differential privacy and find that fairness violations grow non-linearly when systems rely on a chain of multiple Boolean conditions. Similarly, in complex information processing pipelines—such as automated misinformation detection systems—errors at early stages (like check-worthiness or evidence retrieval) cascade into later stages (like verdict prediction), resulting in compounded harms for specific stakeholders (Neumann et al.).
Emergent Biases in Multimodal and Multi Agent Systems
The compositionality failure becomes even more pronounced when moving beyond simple pipelines to integrated multimodal models or multi-agent systems.
In multi-agent environments, system-level properties like fairness cannot be reduced to the sum of individual agent behaviors; a multi-agent system can fail global fairness criteria even if every constituent agent satisfies them in isolation (Burkhard). When developers build systems by integrating multiple pre-trained expert models, the ensemble can inherit and amplify the subtle, latent biases of its underlying components (Puerto et al.). This is a common vulnerability in "plug-and-play" frameworks that combine pre-trained models without explicit retraining, as they lack inherent mechanisms to filter or correct the underlying flaws of their base modules (Chung et al.).
This challenge is starkly illustrated in modern Large Language Models (LLMs) and multimodal systems. As argued by D'Amour et al., combining an individually fair text model with an individually fair image model does not guarantee a fair multimodal output. The complex, non-linear interactions between the underlying representations create entirely new, unforeseen biases that were completely absent in the standalone modules.
Sociotechnical and Market Interactions
Finally, algorithmic components do not operate in a vacuum; they interact with economic mechanisms, market competition, and strategic human behavior.
The authors of Finocchiaro et al. demonstrate that combining a fair predictive model with a fair allocation mechanism can still result in discriminatory outcomes. External forces, such as market competition or strategic user adaptations, disrupt the intended fairness of the components.
For example, in two-sided matching markets (such as matching medical residents to hospitals), standard matching algorithms fail to maintain individual fairness even when provided with fair inputs and preferences (Karni et al.). Because solutions optimized for one isolated domain frequently fail when integrated into broader sociotechnical systems, treating fairness metrics as context-independent guarantees is fundamentally flawed (Dolata et al.).
Shifting to System Level Auditing
The non-compositionality of fairness requires a paradigm shift in how algorithmic systems are designed and evaluated. Rather than relying on isolated component testing or simple checklists, practitioners must adopt a holistic approach that analyzes the entire system architecture, data flows, and feedback loops. Only by evaluating fairness at the global, system level can we hope to detect and mitigate the emergent biases that arise when fair components collide.
Key Foundational Work
The Frontiers of Fairness in Machine Learning (2018) — contributed 3 concept elements to this direction.
Towards Composable Bias Rating of AI Services (2018) — contributed 1 concept elements to this direction.
Liveness and Fairness Properties in Multi-Agent Systems (1993) — contributed 1 concept elements to this direction.