Toward a Holistic Framework for Human-AI Collaboration in Safety-Critical Systems

Ricardo Jorge Gomes de Sousa Bento Bessa, Milad Leyli-Abadi, Mouadh Yagoubi, Daniel Boos, Clark Borst, Alberto Castagna, José del R. Millán, Duarte Dias, Adrian Egli, Andrina Eisenegger, Joost Ellerbroek, Anna Fedorova, Cristina Felix, Anton Fuxjäger, Joaquim Geraldes, Samira Hamouche, Mohamed Hassouna, Sjoerd Kop, Bruno Lemetayer, Giulia Leto, Roman Ließner, Jonas Lundberg, Antoine Marot, Maroua Meddeb, Manuel Meyer, Hélio Sales, Viola Schiaffonati, Manuel Schneider, Irene Sturm, Julia Usher, Herke van Hoof, Jan Viebahn, Toni Wäfler, Giacomo Zanotti, Ricardo ChavarriagaOpenalex Unknown 2026 · DOI · OpenAlex

Abstract

The integration of artificial intelligence (AI) into safety-critical systems, where human operators remain central to decision-making, introduces various challenges that existing AI frameworks struggle to address comprehensively. Key concerns involve designing a socio-technical system that balances AI transparency, trust, and explainability with the imperative for robust and reliable decision-making. Presently, while numerous sector-specific solutions exist, a holistic framework that effectively integrates human expertise with AI capabilities remains absent, leaving critical gaps in system design, deployment, and oversight. This chapter proposes a multidisciplinary conceptual framework to enhance human-AI collaboration in critical infrastructures such as power grids, railways, and air traffic management. The different design steps were guided by the requirements of these industrial domains. The framework combines key design principles that support human cognition, leveraging insights from decision theory, mathematics, and specialized engineering domains to optimize AI-assisted decision-making. Furthermore, it embeds trustworthiness and risk assessment methodologies, using tools such as the Assessment List for Trustworthy Artificial Intelligence (ALTAI) tool to ensure compliance with ethical and regulatory requirements.

Introduction: The Socio-Technical Challenge

As critical infrastructures—such as power grids, railways, and air traffic management—face increasing pressure from decarbonization, digitalization, and rising demand, the complexity of managing these systems is outstripping human cognitive limits. While AI offers a solution, its integration into safety-critical environments is fraught with risk. Existing AI frameworks are often fragmented or purely technical, failing to account for the "socio-technical" reality: human operators must remain legally and operationally responsible for high-stakes decisions.

The authors identify a "critical gap" in how we design, deploy, and oversee systems where humans and AI must collaborate seamlessly under pressure. They tackle the challenge of maintaining operator trust and situational awareness while preventing "deskilling"—the loss of human expertise that occurs when automation takes over too much of the cognitive load.

The Proposed Framework

The authors propose a multidisciplinary, holistic framework for integrating AI into safety-critical infrastructures. By moving beyond sector-specific silos, the framework defines a roadmap that balances AI autonomy with human oversight, ensuring that "human-in-the-loop" is a functional requirement for system resilience.

Conceptual framework methodology and system design overview.

The framework employs a Model-Based System Engineering (MBSE) approach using the ARCADIA method, which structures the system design into hierarchical views: Operational, Functional, Logical, and Physical. Key methodologies integrated include:

  • Cognitive Systems Engineering (CSE): Merging the Joint Control Framework (JCF) for planning with Ecological Interface Design (EID) for constraint visualization.
  • Reinforcement Learning (RL): Utilizing Multi-Agent RL (MARL) for complex coordination and Multi-Objective RL to handle conflicting goals.
  • Hypervision: A novel HMI (Human-Machine Interface) concept designed to synthesize disparate data streams into a unified, task-prioritized interface.
  • ALTAI Adaptation: Adapting the EU’s Assessment List for Trustworthy Artificial Intelligence into a "non-calculative" risk assessment tool.

Key Insights: Beyond Automation

The framework rejects a "one-size-fits-all" approach to automation, defining three distinct levels:

  1. Full Human Control: AI acts as a passive assistant/advisor.
  2. Co-learning: A bidirectional partnership where AI and humans refine their collaboration over time.
  3. Autonomous AI: The system operates independently with humans acting as high-level supervisors.

The paper emphasizes shifting from "automate" to "informate"—empowering human capabilities rather than just replacing human effort. This is crucial for maintaining system resilience during "edge cases" where AI models might fail. Furthermore, in safety-critical systems where historical data is often insufficient for precise probability estimates, the authors argue for a qualitative approach to risk, focusing on reducing "vulnerability" and "exposure" when "hazard" probabilities are uncertain.

Results and Cross-Domain Applicability

The primary result is the technology-agnostic blueprint for developers. Through a similarity analysis, the authors found that while decision impacts are domain-specific, the characteristics of decision-making across power, rail, and air traffic are remarkably similar (40-50%). This justifies the creation of a cross-sector framework. To validate this, the authors identify high-fidelity digital environments: Grid2op for power grid congestion, Flatland for railway scheduling, and Bluesky for air traffic management. The paper also produces an extensive list of operational and human-in-the-loop requirements that serve as a standardized checklist for future projects.

Limitations and Future Work

While grounded in real-world use cases, the framework remains a conceptual prototype. Future work will focus on:

  • Empirical Validation: Rigorous testing within the identified simulators to quantify performance under stress.
  • Mitigating Deskilling: Developing "cognitive forcing" strategies to keep humans mentally engaged even when AI performs well.
  • Regulatory Alignment: Continuous updates to ensure the "Trustworthiness Assessment" module remains compliant with evolving international standards like the EU AI Act.

Summary updated

Abstract

Abstract The integration of artificial intelligence (AI) into safety-critical systems, where human operators remain central to decision-making, introduces various challenges that existing AI frameworks struggle to address comprehensively. Key concerns involve designing a socio-technical system that balances AI transparency, trust, and explainability with the imperative for robust and reliable decision-making. Presently, while numerous sector-specific solutions exist, a holistic framework that effectively integrates human expertise with AI capabilities remains absent, leaving critical gaps in system design, deployment, and oversight. This chapter proposes a multidisciplinary conceptual framework to enhance human-AI collaboration in critical infrastructures such as power grids, railways, and air traffic management. The different design steps were guided by the requirements of these industrial domains. The framework combines key design principles that support human cognition, leveraging insights from decision theory, mathematics, and specialized engineering domains to optimize AI-assisted decision-making. Furthermore, it embeds trustworthiness and risk assessment methodologies, using tools such as the Assessment List for Trustworthy Artificial Intelligence (ALTAI) tool to ensure compliance with ethical and regulatory requirements.
BibTeX
@inproceedings{bessa2026toward,
  author = {Ricardo Jorge Gomes de Sousa Bento Bessa and Milad Leyli-Abadi and Mouadh Yagoubi and Daniel Boos and Clark Borst and Alberto Castagna and José del R. Millán and Duarte Dias and Adrian Egli and Andrina Eisenegger and Joost Ellerbroek and Anna Fedorova and Cristina Felix and Anton Fuxjäger and Joaquim Geraldes and Samira Hamouche and Mohamed Hassouna and Sjoerd Kop and Bruno Lemetayer and Giulia Leto and Roman Ließner and Jonas Lundberg and Antoine Marot and Maroua Meddeb and Manuel Meyer and Hélio Sales and Viola Schiaffonati and Manuel Schneider and Irene Sturm and Julia Usher and Herke van Hoof and Jan Viebahn and Toni Wäfler and Giacomo Zanotti and Daniel Boos and Clark Borst and Ricardo Chavarriaga and Joost Ellerbroek and Roman Liessner and Jonas Lundberg and Antoine Marot and Viola Schiaffonati and Manuel Schneider},
  title = {{Toward a Holistic Framework for Human-AI Collaboration in Safety-Critical Systems}},
  booktitle = {Openalex Unknown},
  year = {2026},
  doi = {10.1007/978-3-032-10561-5\_13},
}

Loading related content...

Figure