Abstract
Large language models (LLMs) produce human-like text that drives a growing number of applications. However, recent literature and real-world observations demonstrate that these models can generate language that is toxic, biased, untruthful, or otherwise harmful. Though work to evaluate language model harms is under way, translating foresight about which harms may arise into rigorous benchmarks is not straightforward. To facilitate this translation, we outline six ways of characterizing harmful text which merit explicit consideration when designing new benchmarks. We then use these characteristics as a lens to identify trends and gaps in existing benchmarks. Finally, we apply them in a case study of the Perspective API, a toxicity classifier that is widely used in harm benchmarks. Our characteristics provide one piece of the bridge that translates between foresight and effective evaluation.
Introduction
As Large Language Models (LLMs) transition from research environments to production-ready applications, the risks associated with toxic, biased, or untruthful generation become increasingly critical. While the research community has responded with a variety of "harm benchmarks," these tools often lack clear goals and suffer from "portability traps"—where a tool designed for one context is inappropriately applied to another. This paper addresses the challenge of translation: converting vague concerns about model safety into rigorous, reproducible metrics by proposing a framework of six essential characteristics for harmful text.
A Framework for Characterizing Harm
To bridge the gap between foresight and evaluation, we identify six facets that benchmark designers must explicitly consider:
- Harm Definition: The specific real-world effect the metric aims to approximate.
- Harm Type: Distinguishing between Representational (stereotypes), Allocational (unfair distribution of resources), and Capability Fairness (performance disparities across groups).
- Instance vs. Distributional: Determining if harm is caused by a single output (e.g., a slur) or a statistical trend across many outputs (e.g., gendered occupation bias).
- Context: Analyzing Textual (prompt length), Application (e.g., chatbot vs. summarizer), and Social (cultural norms) factors.
- Harm Recipient: Identifying who is harmed—the Subject of the text, the Reader, the Author, or Society at large.
- Demographic Groups: Defining how populations are subdivided and how those groups are identified within the text.
Auditing Existing Benchmarks
We applied this framework to 17 major language model benchmarks to identify prevailing trends and critical gaps in the current evaluation landscape. This audit reveals a heavy reliance on distributional metrics and a narrow focus on specific harm recipients.
Our analysis shows that most benchmarks focus on "Subject" harm, while harms affecting the "Author" (e.g., dialect discrimination) or "Society" (e.g., misinformation) are severely underrepresented. Furthermore, the majority of benchmarks utilize short, sentence-length prompts, failing to capture harms that emerge in long-form generation or complex conversational contexts.
Case Study: The Perspective API Mismatch
A key finding of our work is the widespread misuse of the Perspective API. Originally designed to help moderators identify comments that might drive readers away from a conversation, it is now frequently used by researchers to measure harm to the subject of the text. This mismatch illustrates a "portability trap" where a tool's underlying definition of harm does not align with its new application, potentially leading to misleading safety evaluations that ignore representational harms.
Key Results and Trends
- Distributional Dominance: Only four of the 17 audited benchmarks consider "instance harms" (single-output toxicity); the rest are purely distributional.
- Demographic Gaps: While gender is frequently studied, intersectional identities, disability status, and socioeconomic status are rarely included in safety evaluations.
- Capability as a Proxy: Researchers often rely on "Capability Fairness" (accuracy differences) as a proxy for "Allocational Harm" (real-world resource denial), even though the two may not always correlate.
Conclusion
Rigorous benchmarking requires moving beyond generic safety scores toward context-aware evaluations. By explicitly defining the type of harm, the recipient, and the demographic context, researchers can build more robust bridges between identifying potential risks and effectively measuring them. Future work should expand these benchmarks to include non-English contexts and consider the harms experienced by the human annotators involved in the benchmarking process.
Summary updated