Beyond model interpretability: socio-structural explanations in machine learning

Andrew Smart, Atoosa KasirzadehAi soc 2025 · DOI

TL;DR

This paper argues that traditional "model-centric" explanations for machine learning (ML) outputs are insufficient for high-stakes domains. They propose "socio-structural explanations" that consider the broader social and institutional contexts in which ML models operate, demonstrating their necessity with a case study of a biased healthcare algorithm.

What Problem Does This Paper Address?

The field of interpretable ML aims to make opaque ML models understandable. Current approaches primarily focus on "model-centric" explanations, which reveal how the model itself works (e.g., feature importance, saliency maps). However, the authors argue that these methods fall short when ML models are deployed in complex, socially embedded domains like healthcare or hiring. In these contexts, the model's outputs are not solely determined by its internal logic but are also shaped by existing societal structures, biases, and inequalities. Without considering these external factors, model-centric explanations can be misleading or incomplete, failing to address the root causes of problematic outcomes. The paper addresses the gap between understanding how a model works and understanding why its outputs have certain societal impacts.

Methods

The paper's core contribution is the proposal and defense of "socio-structural explanations" as a necessary complement to model-centric interpretability. This approach draws on social philosophy to argue that ML models are not isolated entities but are embedded within and shaped by social structures. Socio-structural explanations aim to illustrate how these broader societal factors contribute to and partially explain ML model outputs.

The authors differentiate between:

  • Model-centric explanations: Focus on the internal workings of the ML model (e.g., feature importance, counterfactuals, mechanistic interpretations).
  • Socio-structural explanations: Focus on the external social and institutional context that influences the model's behavior and outputs.

They illustrate this by analyzing a case study of a racially biased healthcare allocation algorithm, demonstrating how a socio-structural lens reveals the systemic issues that model-centric explanations alone would miss.

Datasets

Not discussed in the paper. The paper focuses on conceptual arguments and a case study analysis rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Model-centric explanations are not enough: In high-stakes, socially embedded domains, understanding how a model works isn't sufficient to understand why it produces certain outcomes.
  • Social structures matter: ML models operate within complex social systems (e.g., historical inequalities, societal norms, institutional practices) that profoundly shape their behavior and impact.
  • Socio-structural explanations reveal root causes: By considering these external factors, we can uncover deeper, systemic reasons for algorithmic bias and unfairness that model-centric approaches miss.
  • Healthcare allocation bias example: The paper uses a real-world example where an algorithm used healthcare costs as a proxy for need, inadvertently discriminating against Black patients due to historical disparities in healthcare access and spending. Socio-structural explanations highlight that lower costs for Black patients reflected systemic inequities, not better health.
  • Beyond interpretability: True transparency in high-stakes ML requires a broader analysis that extends beyond the model itself to its societal context.

Key Results

The paper does not present quantitative results in the traditional sense. Its key "results" are conceptual:

  • A framework for understanding ML interpretability that includes socio-structural explanations.
  • A compelling argument for the necessity of socio-structural explanations in domains with significant societal impact.
  • A detailed analysis of a healthcare allocation algorithm that demonstrates how socio-structural factors explain its racial bias, which would be overlooked by model-centric methods alone.

Limitations and Future Work

  • Challenge of producing socio-structural explanations: The authors acknowledge that generating rigorous socio-structural explanations can be challenging, requiring significant sociological understanding and interdisciplinary expertise.
  • Practitioner preparedness: Many ML practitioners may not feel equipped to analyze and respond to social structures, or may be hindered by constraints in time, training, or resources.
  • Need for integration: The paper calls for further research into integrating socio-structural understanding into all stages of the ML lifecycle, from problem formulation to model monitoring.
  • Broader applicability: While the focus is on ML, the authors note that the importance of socio-structural explanations generalizes to a broader range of automated decision systems.

Summary updated

BibTeX
@article{smart2025beyond,
  author = {Andrew Smart and Atoosa Kasirzadeh},
  title = {{Beyond model interpretability: socio-structural explanations in machine learning}},
  journal = {Ai soc},
  year = {2025},
  doi = {10.1007/s00146-024-02056-1},
}
Figure