From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems

Shalaleh Rismani, Roel Dobbe, AJung Moon

TL;DR

This paper adapts a robust system safety framework, System Theoretic Process Analysis (STPA), to analyze potential harms in AI systems. By extending STPA to encompass the entire AI development and deployment lifecycle, the authors introduce a new guideline (PHASE) that enables a more holistic understanding of AI risks, moving beyond isolated component failures to system-level, sociotechnical hazards.

What Problem Does This Paper Address?

Current approaches to identifying and mitigating harms from AI systems often focus on individual components like training data or models in isolation. This siloed perspective overlooks critical hazards that arise from the interactions between these components, as well as how AI systems are embedded within broader organizational and development processes. The paper addresses this gap by proposing a system-level hazard analysis framework that considers AI as a sociotechnical system, acknowledging that safety is an emergent property of the entire ecosystem, not just its parts.

Methods

The core methodology is the adaptation of System Theoretic Process Analysis (STPA), a well-established framework from safety engineering. STPA takes a control and system-theoretic approach to identify losses, hazards, and unsafe control actions within a system. The authors translate STPA's four key steps – identifying purpose of analysis, creating control diagrams, identifying unsafe control actions, and identifying loss scenarios – to the context of AI systems. They developed a new guideline, Process-oriented Hazard Analysis for AI Systems (PHASE), to facilitate this translation, specifically addressing unique AI traits like model opacity, capability uncertainty, and output complexity. The research involved applying STPA and developing PHASE through three case studies: a predictive algorithm for early warning of neonatal sepsis, an RL-based automated insulin injection system, and text-to-image (T2I) models used in creative practice.

Datasets

Not discussed in the paper. The paper focuses on the application of a framework to existing or developing AI systems rather than training or evaluating models on specific datasets.

Key Insights and Ideas

  • AI as a Sociotechnical System: The paper strongly emphasizes that AI systems are not purely technical artifacts but are deeply embedded in social and organizational contexts. Harms often arise from the interplay of human actors, processes, and the technology itself.
  • Beyond Safety-Critical Harms: Traditional STPA often focuses on catastrophic failures (e.g., loss of life). The authors successfully extend the framework to encompass a broader range of harms, including performance-related and sociotechnical issues like loss of privacy, fairness, and reputation.
  • PHASE as an Enabler: The developed PHASE guideline is crucial for making STPA accessible and applicable to AI practitioners, bridging the gap between safety engineering principles and the unique challenges of ML.
  • Emergent Hazards: AI systems, especially those that learn and adapt, can introduce emergent hazards that are not apparent during initial design. The framework helps in identifying and monitoring these evolving risks.
  • Traceable Accountability: By mapping control structures and causal chains, PHASE enables the creation of traceable accountability chains, clarifying who is responsible for mitigating specific harms.

Key Results

The paper demonstrates that STPA, with adaptations through the PHASE guideline, can be effectively applied to analyze AI systems. The authors identified:

  • Four key affordances of PHASE:
    1. System-level hazard detection: Identifying hazards arising from component interactions and systemic issues.
  • Acknowledgment of social factors: Explicitly incorporating social and ethical considerations into hazard analysis.
  • Traceable accountability chains: Establishing clear links between harms and mitigation responsibilities.
  • Ongoing monitoring: Facilitating continuous assessment and mitigation of evolving AI risks.
  • Case study findings: The three case studies (neonatal sepsis prediction, insulin injection, T2I models) illustrated how STPA/PHASE can uncover specific hazards related to data processing, model development, integration, and end-use, highlighting issues like data imbalance, model interpretability, prompt filtering, and coordination breakdowns.

Limitations and Future Work

  • Generalizability: The findings are based on three case studies, and while chosen to represent diverse AI paradigms, they may not cover the full spectrum of AI applications. Broader application is needed to test the generalizability of the PHASE guideline.
  • Analyst Dependence: The flexibility of STPA/PHASE can be a double-edged sword; the quality of the analysis depends on the analyst's expertise and diligence. The authors suggest cross-validation by multiple analysts to improve robustness.
  • Conceptual Flexibility: While a strength, the broad conceptual flexibility of mapping social and technical elements could be misused if not applied rigorously.
  • Agentic AI: While the framework can map control and feedback, it currently frames interactions as control and feedback, which might not fully capture the nuances of human relationships or highly agentic AI systems.
  • Future Work: The authors suggest further research into applying PHASE to more diverse AI systems, exploring its use in governing increasingly agentic AI, and developing industry standards for quality PHASE analysis. They also propose making PHASE available as an open-source guideline.

Summary updated

Abstract

To effectively address potential harms from AI systems, it is essential to identify and mitigate system-level hazards. Current analysis approaches focus on individual components of an AI system, like training data or models, in isolation, overlooking hazards from component interactions or how they are situated within a company's development process. To this end, we draw from the established field of system safety, which considers safety as an emergent property of the entire system, not just its components. In this work, we translate System Theoretic Process Analysis (STPA) - a recognized system safety framework - for analyzing AI operation and development processes. We focus on systems that rely on machine learning algorithms and conducted STPA on three case studies involving linear regression, reinforcement learning, and transformer-based generative models. Our analysis explored how STPA's control and system-theoretic perspectives apply to AI systems and whether unique AI traits - such as model opacity, capability uncertainty, and output complexity - necessitate significant modifications to the framework. We find that the key concepts and steps of conducting an STPA readily apply, albeit with a few adaptations tailored for AI systems. We present the Process-oriented Hazard Analysis for AI Systems (PHASE) as a guideline that adapts STPA concepts for AI, making STPA-based hazard analysis more accessible. PHASE enables four key affordances for analysts responsible for managing AI system harms: 1) detection of hazards at the systems level, including those from accumulation of disparate issues; 2) explicit acknowledgment of social factors contributing to experiences of algorithmic harms; 3) creation of traceable accountability chains between harms and those who can mitigate the harm; and 4) ongoing monitoring and mitigation of new hazards.
BibTeX
@misc{rismani2024from,
  author = {Shalaleh Rismani and Roel Dobbe and AJung Moon},
  title = {{From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems}},
  year = {2024},
  eprint = {2410.22526},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.22526},
}
Figure