Open Problems in Frontier AI Risk Management

Marta Ziosi, Miro Plueckebaum, Stephen Casper, Henry Papadatos, Ze Shen Chin, Peter Slattery, James Gealy, Tim G. J. Rudner, Brian Tse, Ariel Gil, Patricia Paskov, Maximilian Negele, Rokas Gipiškis, Nada Madkour, Vera Lummis, Rupal Jain, Luise Eder, Kristina Fort, Malou C. van Draanen Glismann, Inès Belhadj, Amin Oueslati, Anna K. Wisakanto, Richard Mallah, Koen Holtman, Ranj Zuhdi, Daniel S. Schiff, Jessica Newman, Malcolm Murray, Robert TragerarXiv 2026 · arXiv

TL;DR

Existing risk management standards are built for narrow, deterministic AI systems and fail to address the unique, fast-evolving challenges of general-purpose "frontier" AI. This paper systematically maps out the open technical and organizational problems across the entire risk management lifecycle, categorizing where the safety community lacks consensus, alignment, or effective implementation. It serves as an agenda-setting framework to guide future safety research, developer practices, and regulatory coordination.

What Problem Does This Paper Address?

Traditional risk management frameworks (such as ISO 31000) are designed for mature industries where system boundaries are clear, behaviors are predictable, and historical data is abundant. Frontier AI systems—characterized by their general-purpose nature, emergent capabilities, and non-linear interactions—break these traditional paradigms.

Currently, emerging frontier AI safety practices are often ad-hoc and misaligned with established risk management structures. Without a cohesive, systematic map of the challenges frontier AI poses to risk management, safety initiatives risk relying on flawed assumptions, generating duplicative work, and creating confusion over which standards apply in which contexts. This paper addresses this fragmentation by surfacing and structuring the open problems across the field.

Methods

The authors adopt a problem-oriented, agenda-setting approach rather than proposing specific technical solutions. They structure their literature review around the five core stages of traditional risk management standards (ISO 31000:2018 and ISO/IEC 23894:2023):

Figure 1. Risk Management Process (adapted from ISO, 2018)

For each stage, the paper identifies outstanding "open problems" and classifies them into three distinct types based on the nature of the challenge:

  1. Lack of scientific or technical consensus: The community does not yet know the technical answer or best methodology.
  2. Misalignment with traditional frameworks: Emerging AI safety practices clash with or undermine established risk management principles.
  3. Shortcomings in implementation: A conceptual consensus exists, but practical execution or enforcement is lacking.

Finally, they map out which specific actors (e.g., upstream/downstream developers, regulators, standards bodies, third-party evaluators) are best positioned to address each problem.

Datasets

As this is a systematic literature review and policy framework paper, it does not introduce or utilize empirical machine learning datasets for training or evaluation.

However, the text highlights and critiques the use of existing industry incident repositories—such as the AI Incident Database (AIID), the OECD AI Incidents Monitor, and the MITRE ATLAS Matrix—noting that their reliance on voluntary reporting limits their representativeness and effectiveness for proactive risk identification.

Key Insights and Ideas

  • Capability vs. Harm Dilemma: Frontier AI risk criteria currently rely heavily on "capability thresholds" (what a model can do) as a proxy for risk. This approach primarily captures the possibility of harm while neglecting critical contextual factors like the external threat landscape and deployment configurations.
  • The Shifting Baseline ("Boiling Frog") Effect: Instead of absolute risk, developers frequently measure "marginal risk" relative to a shifting baseline (e.g., a competitor's model, human performance, or their own previous model). This lack of a stable baseline can mask a dangerous accumulation of absolute, ecosystem-level risk.
  • The Illusion of Independent Audits: Many external model evaluations are actually hybrid setups where the developer funds the auditor, defines the evaluation scope, and retains veto power over publishing negative results. This severely undermines structural independence.
  • Fragility and Reversibility of Mitigations: Model-level mitigations like machine unlearning or reinforcement learning from human feedback (RLHF) are often fragile. Adversarial attacks (jailbreaks) can easily reverse these safety guardrails, creating a false sense of security for deployment decisions.

Key Results

The paper's primary output is a structured taxonomy of open problems across the five risk management stages. Key highlights include:

  • Risk Planning: Surfaced the challenge of defining risk boundaries for modular, vendor-integrated systems where AI components are reused across entirely different sectors, weakening downstream accountability.
  • Risk Identification: Highlighted the failure of current techniques (like HAZOP or SWIFT) to account for complex, non-linear interactions and unknown risks that lack historical precedent.
  • Risk Analysis: Identified a critical gap in methodologies for combining heterogeneous evidence streams (e.g., capability evaluations, API usage logs, and human uplift studies) into unified, quantitative risk models.
  • Risk Evaluation: Highlighted the difficulty of aggregating multiple individual risks to judge a system's total risk profile, especially when risks span completely incommensurable domains (e.g., economic cyber-attacks vs. diffuse societal erosion of trust).
  • Risk Mitigation: Documented the lack of shared definitions for what constitutes a "reportable serious incident," leading to fragmented reporting and potential regulatory overload for oversight bodies.

Limitations and Future Work

The authors explicitly note that their surfaced list of open problems is illustrative rather than completely exhaustive. To keep the scope manageable, they intentionally excluded transversal risk governance aspects, such as Communication and Consultation, Monitoring and Review, and Recording and Reporting, though they plan to integrate these into future iterations. Additionally, the paper focuses strictly on safety risks, leaving broader security-centric risk management out of scope for this version.

To ensure the work remains actionable, the authors have complemented the paper with a living online repository designed to continuously track these open problems, minimize research duplication, and foster alignment between traditional risk engineering and frontier AI governance.

Summary updated

Abstract

Frontier AI both amplifies existing risks and introduces qualitatively novel challenges. Not only is there a notable lack of stable scientific consensus resulting from the rapid pace of technological change, but emerging frontier AI safety practices are often misaligned with, or may undermine, established risk management frameworks. To address these challenges, we systematically surface open problems in frontier AI risk management. Adopting a problem-oriented approach, we examine each stage of the risk management process - risk planning, identification, analysis, evaluation, and mitigation - through a structured review of the literature, identifying unresolved challenges and the actors best positioned to address them. Recognising that different types of open problems call for different responses, we classify open problems according to whether they reflect (a) a lack of scientific or technical consensus, (b) misalignment with, or challenges to, established risk management frameworks, or (c) shortcomings in implementation despite apparent consensus and alignment. By mapping these open problems and identifying the actors best positioned to address them - including developers, deployers, regulators, standards bodies, researchers, and third-party evaluators - this work aims to clarify where progress is needed to enable robust and meaningful consensus on frontier AI risk management.The paper does not propose specific solutions; instead, it provides a problem-oriented, agenda-setting reference document, complemented by a living online repository, intended to support coordination, reduce duplication, and guide future research and governance efforts.
BibTeX
@misc{ziosi2026open,
  author = {Marta Ziosi and Miro Plueckebaum and Stephen Casper and Henry Papadatos and Ze Shen Chin and Peter Slattery and James Gealy and Tim G. J. Rudner and Brian Tse and Ariel Gil and Patricia Paskov and Maximilian Negele and Rokas Gipiškis and Nada Madkour and Vera Lummis and Rupal Jain and Luise Eder and Kristina Fort and Malou C. van Draanen Glismann and Inès Belhadj and Amin Oueslati and Anna K. Wisakanto and Richard Mallah and Koen Holtman and Ranj Zuhdi and Daniel S. Schiff and Jessica Newman and Malcolm Murray and Robert Trager},
  title = {{Open Problems in Frontier AI Risk Management}},
  year = {2026},
  eprint = {2604.25982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.25982},
}

Loading related content...

Figure