Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms

Shalaleh Rismani, Renee Shelby, Leah Davis, Negar Rostamzadeh, AJung Moon

This paper, recently published by researchers from McGill University and Google Research, takes a critical look at the "measurement problem" in AI ethics. While the field of Responsible AI (RAI) has produced hundreds of metrics, the authors argue that we are currently "measuring in pieces"—focusing on isolated technical components while losing sight of how harms actually emerge in the real world.

TL;DR

The authors analyzed 791 AI ethics measures from 257 academic papers to reveal a massive "sociotechnical gap." Current evaluations are over-indexed on model-specific metrics for fairness and privacy, but they largely ignore system-wide interactions, lack clear thresholds for what constitutes a "hazard," and fail to connect technical numbers to the lived experiences of human harm.

What Problem Does This Paper Address?

As AI systems move into high-stakes domains like healthcare and criminal justice, we need more than just "accuracy" scores; we need to know if they are ethical. However, the current ecosystem of RAI measures is fragmented.

The authors identify two major failings:

  1. Construct Validity: Many measures don't actually capture the normative qualities (like "trust" or "fairness") they claim to measure.
  2. Modular Blindness: Most metrics focus on a single component—like a dataset or a model's weights—in isolation. In safety engineering, safety is an emergent property of a whole system. You can have a "fair" model that still produces "unfair" outcomes because of how it interacts with a specific user interface or organizational process.

The paper seeks to bridge this gap by applying a System Safety lens to AI ethics, mapping existing measures to a chain of: System Component $\rightarrow$ Attribute $\rightarrow$ Measure $\rightarrow$ Hazard $\rightarrow$ Harm.

Methods

The researchers conducted a massive scoping review of computing literature from 2017 to 2023. Their methodology involved:

  • Identification: Screening over 1,600 papers to find 257 that proposed specific quantitative or qualitative measures for 11 AI ethics principles (based on the Jobin et al. taxonomy).
  • Reflexive Analysis: They extracted 791 individual measures and manually coded them based on what system component they target (Data, Model, Output, User Interaction, or Full System) and what type of harm they signal.
  • Harm Taxonomy: They mapped these measures to the five categories of sociotechnical harm defined by Shelby et al.: Allocative, Representational, Quality of Service, Interpersonal, and Social System harms.

Datasets

The primary "dataset" in this paper is the corpus of 791 RAI measures themselves. The authors have compiled these into a structured dataset (available on GitHub) and created an interactive Sunburst visualization tool to help researchers and practitioners explore which ethics principles are over- or under-measured.

Key Insights and Ideas

  • The "Big Four" Hegemony: 90% of all measures focus on just four principles: Fairness, Transparency, Privacy, and Trust. Principles like "Dignity," "Solidarity," and "Sustainability" are almost entirely ignored in the technical literature.
  • The Model-Centric Bias: Over 57% of measures target the model itself, while only 25% look at the interaction between the user and the output. This suggests we are better at measuring "math" than "impact."
  • The Threshold Vacuum: Only 26.9% of the measures were accompanied by an explicit criterion or threshold (e.g., "the 80% rule" for disparate impact). Without these thresholds, developers have no "red line" to tell them when a system state becomes a hazardous one.
  • Temporal Blindness: Harms often accumulate over time (e.g., the long-term erosion of social trust), yet almost none of the reviewed measures were designed for long-term monitoring. Most are "one-and-done" snapshots taken during development.

Key Results

  • Fairness is the most measured principle (46.4%), primarily linked to Representational and Allocative harms.
  • Privacy measures are almost exclusively linked to Interpersonal harms (95.5%).
  • Sustainability is the rare exception that looks at the Full System (e.g., total carbon footprint), rather than just a single component.
  • Perception-based measures (how a user feels about a system) make up 17.3% of the corpus but are often disconnected from technical performance metrics, making it hard to use them for safety-critical interventions.

Limitations and Future Work

The authors acknowledge that their review is a "snapshot" of a rapidly evolving field; the explosion of Generative AI and LLMs since late 2023 likely introduces new hazards not fully captured here. Furthermore, the coding process is inherently reflexive and subjective.

Future Work: The authors call for a shift toward system-spanning evaluations. Instead of checking if a model is "fair" in a vacuum, we need benchmarks that trace how sensitive inputs influence model decisions and eventually affect human outcomes. They also emphasize the need for "time-aware" evaluation to detect emergent hazards before they turn into irreversible harms.

Summary updated

Abstract

Over the past decade, an ecosystem of measures has emerged to evaluate the social and ethical implications of AI systems, largely shaped by high-level ethics principles. These measures are developed and used in fragmented ways, without adequate attention to how they are situated in AI systems. In this paper, we examine how existing measures used in the computing literature map to AI system components, attributes, hazards, and harms. Our analysis draws on a scoping review resulting in nearly 800 measures corresponding to 11 AI ethics principles. We find that most measures focus on four principles – fairness, transparency, privacy, and trust – and primarily assess model or output system components. Few measures account for interactions across system elements, and only a narrow set of hazards is typically considered for each harm type. Many measures are disconnected from where harm is experienced and lack guidance for setting meaningful thresholds. These patterns reveal how current evaluation practices remain fragmented, measuring in pieces rather than capturing how harms emerge across systems. Framing measures with respect to system attributes, hazards, and harms can strengthen regulatory oversight, support actionable practices in industry, and ground future research in systems-level understanding.
BibTeX
@inproceedings{rismani2025measuring,
  author = {Shalaleh Rismani and Renee Shelby and Leah Davis and Negar Rostamzadeh and AJung Moon},
  title = {{Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aies.v8i3.36706},
}

Loading related content...

Figure