Introduction
As Text-to-Image (T2I) models like DALL-E and Stable Diffusion transition from research curiosities to professional tools, the industry faces a critical challenge: identifying and mitigating social and ethical risks. Traditional machine learning evaluation often focuses on narrow fairness metrics or model-centric benchmarks. However, many harms emerge not from the model’s internal weights, but from the complex sociotechnical systems surrounding it—including data curation processes, organizational dynamics, and user interactions. This paper demonstrates how decades-old safety engineering frameworks can be repurposed to identify these risks before they reach the public.
Methodology: Repurposing Safety Engineering
The researchers conducted a case study applying two well-established safety engineering frameworks to the T2I development lifecycle:
- Failure Mode and Effects Analysis (FMEA): A bottom-up approach that breaks a system into discrete steps to identify potential failures, their effects, and their likelihood.
- System Theoretic Process Analysis (STPA): A top-down approach that views safety as a control problem. It maps "control loops" between different actors—such as Responsible ML teams, product managers, and the model itself—to identify "Unsafe Control Actions."
To ground this analysis in real-world practice, the authors gathered qualitative data through workshops with 15 professional artists and interviews with 8 industry experts, including developers and managers. These sessions helped map the actual workflows and organizational structures used in modern T2I development.
Key Insights: The Sociotechnical Lens
Safety Engineering is Model-Agnostic
A primary finding of the study is that these frameworks remain effective even when the underlying model is a "black box." Because safety engineering focuses on processes and interactions, researchers can identify hazards—such as a communication breakdown between legal and engineering teams—without needing to inspect the model's internal architecture.
The "Erasure" of Neologisms
By applying FMEA to the data processing stage (using the Parti model’s data card as a reference), the authors discovered that aggressive non-English word filters intended to block "adult content" often inadvertently filter out new slang or culturally specific terms (neologisms). This process-level failure leads to the social harm of cultural alienation, a risk that would be missed by simply evaluating the final model's output.
Organizational and Professional Hazards
STPA revealed that many risks are rooted in management and timing. For example, if a Responsible ML team provides safety requirements too late in the development cycle, the product team may skip them to meet a launch deadline. Furthermore, the study examined the "Artist-Client Loop," finding that if a T2I tool cannot accurately represent a specific cultural concept, it creates professional and reputational risks for the artist using the tool, rather than just a "low-quality" image.
Results and Findings
The authors identified over 50 potential hazards and failures across three distinct scopes:
- Scope 1 (Data Processing): 22 failure modes, including issues where over-filtering leads to a lack of diversity in the training set.
- Scope 2 (Model/Product Integration): 14 failure modes and 16 Unsafe Control Actions, such as inappropriate filter thresholds for prompt classifiers.
- Scope 3 (End Use): 11 failure modes and 9 Unsafe Control Actions, highlighting how miscommunication between a client and an artist regarding AI-generated storyboards can lead to harmful content dissemination.
These results demonstrate that safety engineering successfully bridges the gap between technical "bugs" and broader sociotechnical harms.
Limitations and Future Directions
While effective, these frameworks are not without challenges. The analysis is highly dependent on how the analyst "scopes" the system; if a specific team or process is omitted from the map, its associated hazards remain invisible. Additionally, scoring severity and likelihood remains subjective, influenced by the analyst's own background.
Implementing these frameworks requires significant organizational buy-in, as they are time-intensive and require a culture that prioritizes safety over rapid deployment. The authors suggest that the Fairness, Accountability, Transparency, and Ethics (FATE) community should work toward creating standardized "failure mode libraries" specifically for common AI components to streamline this process in the future.
Summary updated