TL;DR
Existing risk management standards are built for narrow, deterministic AI systems and fail to address the unique, fast-evolving challenges of general-purpose "frontier" AI. This paper systematically maps out the open technical and organizational problems across the entire risk management lifecycle, categorizing where the safety community lacks consensus, alignment, or effective implementation. It serves as an agenda-setting framework to guide future safety research, developer practices, and regulatory coordination.
What Problem Does This Paper Address?
Traditional risk management frameworks (such as ISO 31000) are designed for mature industries where system boundaries are clear, behaviors are predictable, and historical data is abundant. Frontier AI systems—characterized by their general-purpose nature, emergent capabilities, and non-linear interactions—break these traditional paradigms.
Currently, emerging frontier AI safety practices are often ad-hoc and misaligned with established risk management structures. Without a cohesive, systematic map of the challenges frontier AI poses to risk management, safety initiatives risk relying on flawed assumptions, generating duplicative work, and creating confusion over which standards apply in which contexts. This paper addresses this fragmentation by surfacing and structuring the open problems across the field.
Methods
The authors adopt a problem-oriented, agenda-setting approach rather than proposing specific technical solutions. They structure their literature review around the five core stages of traditional risk management standards (ISO 31000:2018 and ISO/IEC 23894:2023):
For each stage, the paper identifies outstanding "open problems" and classifies them into three distinct types based on the nature of the challenge:
- Lack of scientific or technical consensus: The community does not yet know the technical answer or best methodology.
- Misalignment with traditional frameworks: Emerging AI safety practices clash with or undermine established risk management principles.
- Shortcomings in implementation: A conceptual consensus exists, but practical execution or enforcement is lacking.
Finally, they map out which specific actors (e.g., upstream/downstream developers, regulators, standards bodies, third-party evaluators) are best positioned to address each problem.
Datasets
As this is a systematic literature review and policy framework paper, it does not introduce or utilize empirical machine learning datasets for training or evaluation.
However, the text highlights and critiques the use of existing industry incident repositories—such as the AI Incident Database (AIID), the OECD AI Incidents Monitor, and the MITRE ATLAS Matrix—noting that their reliance on voluntary reporting limits their representativeness and effectiveness for proactive risk identification.
Key Insights and Ideas
- Capability vs. Harm Dilemma: Frontier AI risk criteria currently rely heavily on "capability thresholds" (what a model can do) as a proxy for risk. This approach primarily captures the possibility of harm while neglecting critical contextual factors like the external threat landscape and deployment configurations.
- The Shifting Baseline ("Boiling Frog") Effect: Instead of absolute risk, developers frequently measure "marginal risk" relative to a shifting baseline (e.g., a competitor's model, human performance, or their own previous model). This lack of a stable baseline can mask a dangerous accumulation of absolute, ecosystem-level risk.
- The Illusion of Independent Audits: Many external model evaluations are actually hybrid setups where the developer funds the auditor, defines the evaluation scope, and retains veto power over publishing negative results. This severely undermines structural independence.
- Fragility and Reversibility of Mitigations: Model-level mitigations like machine unlearning or reinforcement learning from human feedback (RLHF) are often fragile. Adversarial attacks (jailbreaks) can easily reverse these safety guardrails, creating a false sense of security for deployment decisions.
Key Results
The paper's primary output is a structured taxonomy of open problems across the five risk management stages. Key highlights include:
- Risk Planning: Surfaced the challenge of defining risk boundaries for modular, vendor-integrated systems where AI components are reused across entirely different sectors, weakening downstream accountability.
- Risk Identification: Highlighted the failure of current techniques (like HAZOP or SWIFT) to account for complex, non-linear interactions and unknown risks that lack historical precedent.
- Risk Analysis: Identified a critical gap in methodologies for combining heterogeneous evidence streams (e.g., capability evaluations, API usage logs, and human uplift studies) into unified, quantitative risk models.
- Risk Evaluation: Highlighted the difficulty of aggregating multiple individual risks to judge a system's total risk profile, especially when risks span completely incommensurable domains (e.g., economic cyber-attacks vs. diffuse societal erosion of trust).
- Risk Mitigation: Documented the lack of shared definitions for what constitutes a "reportable serious incident," leading to fragmented reporting and potential regulatory overload for oversight bodies.
Limitations and Future Work
The authors explicitly note that their surfaced list of open problems is illustrative rather than completely exhaustive. To keep the scope manageable, they intentionally excluded transversal risk governance aspects, such as Communication and Consultation, Monitoring and Review, and Recording and Reporting, though they plan to integrate these into future iterations. Additionally, the paper focuses strictly on safety risks, leaving broader security-centric risk management out of scope for this version.
To ensure the work remains actionable, the authors have complemented the paper with a living online repository designed to continuously track these open problems, minimize research duplication, and foster alignment between traditional risk engineering and frontier AI governance.
Summary updated