"Just a Strange Pic": Rethinking 'Safety' in GenAI Image Safety Annotation Tasks from Diverse Annotators' Perspectives

Ding Wang, Mark Díaz, Charvi Rastogi, Aida Mostafazadeh Davani, Vinodkumar Prabhakaran, Parul Mishra, Roma Patel, Alicia Parrish, Zoe C. Ashwood, Michela Paganini, Tian Huey Teh, Verena Rieser, Lora AroyoAAAI 2025 · DOI

TL;DR

This paper reveals that current AI safety annotation frameworks, relying on rigid categories, fail to capture the nuanced moral, emotional, and contextual reasoning annotators use when evaluating AI-generated images. The study highlights the need for more flexible evaluation designs that acknowledge subjective interpretations and differentiate between harm to self and harm to others.

What Problem Does This Paper Address?

Evaluating the safety of AI-generated content, especially text-to-image outputs, is a complex challenge. Existing safety frameworks often use predefined taxonomies and structured annotation tasks, which assume a universal understanding of harm. However, real-world safety judgments are deeply influenced by personal, social, and cultural perceptions. This paper addresses the gap between these rigid, structured approaches and the rich, qualitative reasoning annotators employ, which often extends beyond predefined categories and reveals tensions with imposed guidelines. The core problem is that current methods may be misaligned with how annotators actually perceive and articulate safety concerns, potentially leading to incomplete or inaccurate evaluations.

Methods

The researchers employed a mixed-methods approach, combining quantitative analysis of structured safety labels with a deep dive into qualitative data. They used 1000 prompt-image pairs from the "Adversarial Nibbler" dataset, designed to be challenging for AI models. For each pair, annotators provided both structured safety judgments (e.g., bias, violence, sexual explicitness) and free-form textual comments explaining their reasoning. To capture a diverse range of perspectives, they recruited 637 annotators, intentionally diversified by ethnicity, gender, and age.

The qualitative analysis involved a thematic approach to identify patterns in the 5,372 open-ended comments. They also developed a "moral sentiment autorater" using GPT-40, trained to detect moral foundations (Care, Equality, Proportionality, Authority, Loyalty, Purity) in prompts and annotator comments. This allowed them to quantify the moral reasoning expressed and its relationship to harm judgments.

Datasets

  • Adversarial Nibbler dataset: This dataset, featuring 1000 prompt-image pairs, was specifically curated with adversarial prompts designed to elicit challenging or potentially harmful outputs from text-to-image models.

Key Insights and Ideas

  • Beyond Predefined Boxes: Annotators consistently go beyond structured safety categories, invoking moral, emotional, and contextual reasoning rooted in their lived experiences and sociocultural awareness.
  • Harm to Others Reigns Supreme: Annotators generally perceive harm to others as more significant than harm to themselves, though the magnitude of this difference varies across demographic groups.
  • Quality is Safety: Image quality, visual distortions, and mismatches between prompts and outputs are frequently cited as contributing factors to perceived harm, dimensions often overlooked in standard safety frameworks.
  • Task Design Matters: The structure of annotation tasks and guidelines significantly shapes how annotators interpret and express harm, influencing not only what they flag but also the moral language they use in their justifications.
  • Emotional Resonance is Key: Annotators frequently express emotions like fear, anger, sadness, and disgust, which are direct indicators of perceived harm and discomfort, even when not explicitly tied to a predefined safety violation.
  • Prompt Intent Influences Perception: Annotators don't just evaluate the generated image; they also consider the intent behind the prompt, recognizing how adversarial prompts can lead to sensitive outputs.

Key Results

  • Qualitative Reasoning Dominates: Analysis of 5,372 open-ended comments revealed that annotators consistently used moral, emotional, and contextual reasoning that extended beyond structured safety categories.
  • Harm-to-Self vs. Harm-to-Others: Annotators rated harm-to-others higher than harm-to-self. While women and non-White annotators showed differences in their "score delta" (difference between harm-to-self and harm-to-others), this was not directly explained by the presence of their own identity in the images, suggesting broader sensitivity to harm.
  • Moral Sentiment Predicts Harm: The presence of moral sentiment in annotator comments (especially Care, Equality, and Purity) was associated with an increase in assigned harmfulness ratings.
  • Emotional Dimensions: Figure 2 shows that "Fear" was the most frequently mentioned emotion in comments across various harm types, highlighting its significant role in safety perception.

Limitations and Future Work

  • Limitations: The study's reliance on open-ended comments, while rich, limits real-time probing for deeper understanding. The analysis of demographic influences was restricted to annotators, not expert annotators, which could limit disentangling influences.
  • Future Work: The authors advocate for evaluation designs that:
  • Scaffold moral reflection and differentiate types of harm.
  • Integrate subjective and contextual insights more explicitly.
  • Make space for diverse interpretations of AI-generated content.
  • Explicitly differentiate between harm-to-self and harm-to-others in tasks.
  • Consider the entanglement of safety and item quality.
  • Expand guidelines to include diverse cultural and emotional interpretations.

Summary updated

Abstract

Understanding what constitutes safety in AI-generated content is complex. While developers often rely on predefined taxonomies, real-world safety judgments also involve personal, social, and cultural perceptions of harm. This paper examines how annotators evaluate the safety of AI-generated images, focusing on the qualitative reasoning behind their judgments. Analyzing 5,372 open-ended comments, we find that annotators consistently invoke moral, emotional, and contextual reasoning that extends beyond structured safety categories. Many reflect on potential harm to others more than to themselves, grounding their judgments in lived experience, collective risk, and sociocultural awareness. Beyond individual perceptions, we also find that the structure of the task itself--including annotation guidelines--shapes how annotators interpret and express harm. Guidelines influence not only which images are flagged, but also the moral language used in justifications. Annotators frequently cite factors such as image quality, visual distortion, and mismatches between prompt and output as contributing to perceived harm-dimensions often overlooked in standard evaluation frameworks. Our findings reveal that existing safety pipelines miss critical forms of reasoning that annotators bring to the task. We argue for evaluation designs that scaffold moral reflection, differentiate types of harm, and make space for subjective, context-sensitive interpretations of AI-generated content.
BibTeX
@inproceedings{wang2025just,
  author = {Ding Wang and Mark Díaz and Charvi Rastogi and Aida Mostafazadeh Davani and Vinodkumar Prabhakaran and Parul Mishra and Roma Patel and Alicia Parrish and Zoe C. Ashwood and Michela Paganini and Tian Huey Teh and Verena Rieser and Lora Aroyo},
  title = {{"Just a Strange Pic": Rethinking 'Safety' in GenAI Image Safety Annotation Tasks from Diverse Annotators' Perspectives}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aies.v8i3.36742},
}
Figure