Towards AI Safety: A Taxonomy for AI System Evaluation

Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing

Abstract

The advent of advanced AI underscores the urgent need for comprehensive safety evaluations, necessitating collaboration across communities (i.e., AI, software engineering, and governance). However, divergent practices and terminologies across these communities, combined with the complexity of AI systems—of which models are only a part—and environmental affordances (e.g., access to tools), obstruct effective communication and comprehensive evaluation. This paper proposes a framework for AI system evaluation comprising three components: 1) harmonised terminology to facilitate communication across communities involved in AI safety evaluation; 2) a taxonomy identifying essential elements for AI system evaluation; 3) a mapping between AI lifecycle, stakeholders, and requisite evaluations for accountable AI supply chain. This framework catalyses a deeper discourse on AI system evaluation beyond model-centric approaches.

Introduction: The Fragmented Landscape of AI Evaluation

Currently, AI evaluation is fragmented. AI researchers focus on model alignment, software engineers focus on system quality, and policy-makers focus on societal impact. These groups often use the same words (like "evaluation" or "testing") to mean different things, leading to communication breakdowns.

Furthermore, most safety evaluations are model-centric. However, a safe model can become unsafe when integrated into a system with poor user interfaces, lack of guardrails, or access to external tools (environmental affordances). For example, an object detection model might be 99% accurate, but if the autonomous vehicle's braking system has high latency, the system is still unsafe. This paper proposes a conceptual shift: evaluating the entire "body" and "environment" (the system) rather than just the "brain" (the model).

The AI System Evaluation Framework

The proposed framework consists of three main pillars designed to foster collaboration and accountability across the AI ecosystem.

1. Harmonized Terminology

To resolve communication breakdowns, the framework establishes a shared vocabulary. By synthesizing definitions from AI, software engineering, and governance, it provides clarity on critical concepts. For instance, "Accuracy" is expanded beyond statistical metrics to encompass "task completion fidelity," while terms like "Verification" and "Validation" are clearly delineated to align with established engineering practices.

2. Evaluation Taxonomy

The framework introduces a hierarchical structure that differentiates between Narrow AI (task-specific) and General AI (versatile) across both component and system levels.

Figure 1: Taxonomy for AI system evaluation, distinguishing between component-level and system-level assessments for Narrow and General AI.

As illustrated in Figure 1, evaluation is broken down into:

  • Component-level: Focusing on Data, Models, and Safety Guardrails.
  • System-level: Focusing on Quality/Risk, Accuracy/Correctness, and Capability.

3. Lifecycle and Stakeholder Mapping

Safety is a continuous process rather than a single event. The framework maps specific evaluation activities to the AI lifecycle and identifies the stakeholders responsible for each stage.

Figure 2: Mapping of the AI lifecycle, evaluation types, and responsible stakeholders to ensure accountability across the supply chain.

Figure 2 highlights how "Risk Assessment" and "Impact Assessment" are continuous processes. It assigns specific safety responsibilities to every stakeholder in the AI supply chain, including Producers, Providers, Deployers, and Auditors, ensuring accountability from design to decommissioning.

Key Insights: Beyond the Model-Centric View

  • Environmental Affordances: AI components often live inside deterministic software. Evaluation must account for the tools and APIs the AI can access, which can radically change its safety profile.
  • Safety Guardrails as First-Class Citizens: The paper treats safety guardrails (like red-teaming or output filters) as specific components that require their own evaluation for generalizability and interpretability.
  • Capability vs. Functionality: For General AI (like LLMs), the authors distinguish between functional tasks (what it's told to do) and emergent capabilities (what it can do, including dangerous or unplanned behaviors).
  • Supply Chain Accountability: By mapping evaluations to stakeholders, the paper highlights that AI Producers cannot be the only ones responsible for safety; AI Deployers must perform context-specific system evaluations when integrating models into specific environments.

Conclusion and Future Outlook

The transition from model-centric to system-level evaluation is essential for the safe deployment of advanced AI. While this framework provides a high-level roadmap, implementing it requires significant organizational change and the development of more granular, automated tools. Future work will focus on developing an "integrated evaluation framework" that provides actionable metrics for each stage of the lifecycle mapped out in this study.

Summary updated

Abstract

The advent of advanced AI underscores the urgent need for comprehensive safety evaluations, necessitating collaboration across communities (i.e., AI, software engineering, and governance). However, divergent practices and terminologies across these communities, combined with the complexity of AI systems-of which models are only a part-and environmental affordances (e.g., access to tools), obstruct effective communication and comprehensive evaluation. This paper proposes a framework for AI system evaluation comprising three components: 1) harmonised terminology to facilitate communication across communities involved in AI safety evaluation; 2) a taxonomy identifying essential elements for AI system evaluation; 3) a mapping between AI lifecycle, stakeholders, and requisite evaluations for accountable AI supply chain. This framework catalyses a deeper discourse on AI system evaluation beyond model-centric approaches.
BibTeX
@inproceedings{xia2024towards,
  author = {Boming Xia and Qinghua Lu and Liming Zhu and Zhenchang Xing},
  title = {{Towards AI Safety: A Taxonomy for AI System Evaluation}},
  booktitle = {AIware},
  year = {2024},
  doi = {10.1145/3664646.3664766},
}
Figure