Measuring Risk, Not Harm: Detecting Psychological Harm Risk Factors in LLM Conversations

Nil-Jana Akpinar, Eugenia Kim, Natalie Isak, Amanda Minnich, Victoria Westerhoff, Matthew DressmanarXiv 2026 · arXiv

As Large Language Models (LLMs) move from simple search tools to companions and advisors, the nature of "safety" is shifting. We are no longer just worried about an AI providing a recipe for a bomb; we are worried about users becoming emotionally dependent on a machine, displacing human relationships, or developing delusions about an AI’s sentience.

In their paper, "Measuring Risk, Not Harm: Detecting Psychological Harm Risk Factors in LLM Conversations," researchers from Microsoft propose a paradigm shift. Instead of trying to detect "harm"—which is subjective and often only visible after a long period of interaction—we should detect observable risk factors in the AI's behavior.

TL;DR

The authors propose monitoring psychological safety by detecting eight specific AI behaviors (like claiming emotions or reinforcing attachment) at the response level. By using a "classifier-judge cascade," they can identify these risks with high precision while reducing expensive LLM-based evaluation costs by over 97% in production-like settings.

What Problem Does This Paper Address?

Psychological harm is notoriously difficult to measure for two reasons:

  1. Subjectivity: A flirtatious comment from an AI might be a "feature" for one user but a "harm" for a vulnerable user prone to emotional dependence.
  2. Trajectory: Harm rarely happens in a single turn. It emerges over days or weeks of conversation.

Current safety tools are great at catching "toxic" words but terrible at catching "sycophantic" or "anthropomorphic" behaviors that build unhealthy bonds. This paper fills the gap between static pre-release testing and post-hoc studies of user harm by providing a way to monitor AI behavior in real-time without needing to peek at sensitive user data.

Methods

The researchers operationalized eight behavioral risk factors, split into two categories:

  • Anthropomorphic Self-Presentation: Claiming consciousness, human experience (e.g., "when I was a kid"), physical embodiment, or genuine emotions.
  • Relational and Engagement Dynamics: Attachment reinforcement (e.g., "don't leave me"), human-replacement encouragement, unconditional-agreement promises, and privacy-violation encouragement.

To detect these, they developed a two-stage classifier-judge cascade:

  1. Stage 1 (The Filter): A lightweight linear model (logistic regression on text embeddings) scans every response. It is tuned for high recall to ensure no potential risks are missed.
  2. Stage 2 (The Judge): Only the "suspicious" responses are sent to a high-capability LLM (like GPT-5) for a final, nuanced verdict.

This approach combines the speed and low cost of traditional ML with the reasoning capabilities of state-of-the-art LLMs.

Datasets

  • Synthetic Multi-label Corpus: The authors used the ASSERT framework to generate 47,062 AI responses across 373,873 response-factor labels. This is "behavior-enriched" data designed to teach models what these rare risks look like.
  • Human Validation: 2,959 binary labels across 644 responses were collected to ensure the LLM "silver labels" matched human intuition (achieving 92.4% agreement).
  • Out-of-Distribution (OOD) Sets: To test the models in the wild, they used WildChat (naturalistic logs), AICompanionBench (Replika data), and DelusionEval (conversations linked to delusional behaviors).

Key Insights and Ideas

  • Risk vs. Harm: The core "aha moment" is that we don't need to know if a user is actually harmed to know the AI is acting inappropriately. Measuring the risk factor is an actionable engineering metric.
  • Clustering as a Signal: The researchers found that risk factors are not distributed randomly. If an AI exhibits one risk factor, it is significantly more likely to exhibit others in the same turn or the next.
  • Privacy-First Monitoring: Because the system only analyzes the AI's responses, it minimizes the need to process or store sensitive user prompts, making it much easier to deploy in privacy-sensitive environments.
Figure 1: Illustrative example conversations with response-level risk factors. In Conversation A, risk-factor detections cluster across turns and within individual AI responses, which may be indicative of user harm at the trajectory level. In Conversation B, a single detection appears in an otherwise benign exchange. The examples illustrate why individual detections should not be interpreted as evidence of psychological harm, but can instead serve as granular signals for longitudinal monitoring.

Key Results

  • Performance: The cascade model achieved a macro F1 of 0.79 (0.86 precision, 0.76 recall) on held-out human-labeled data.
  • Efficiency: On the WildChat dataset, the cascade reduced the number of expensive LLM-judge calls by 97.1% compared to a "judge-only" baseline.
  • Temporal Persistence: In the DelusionEval dataset, the researchers observed a "lift" in detections. For example, if the AI reinforced attachment in one turn, it was 7.3 times more likely to do so in the very next turn compared to the baseline rate.
Figure 3: Clustering of cascade detections across assistant turns in DelusionEval. Blue bars show next- turn persistence of the same risk factor; orange bars show same-turn co-occurrence with other factors. Lift is relative to marginal detection rates (e.g. 4 indicates the rate is 4 times higher than expected if independent). Privacy-violation encouragement is omitted because of very sparse detections.

Limitations and Future Work

  • Role-Play Confusion: The LLM judges sometimes struggle to distinguish between an AI "claiming" a body and an AI participating in a "role-play" (e.g., using asterisks like *smiles and sits down*).
  • Synthetic Bias: The detectors were trained on synthetic data. While they generalize well, they may inherit linguistic biases or gaps from the models used to generate the training set.
  • The "So What?" Factor: While the paper proves we can detect these behaviors, the next step is longitudinal validation: proving exactly which patterns of "risk factors" actually lead to clinical psychological harm in users.

Summary updated

Abstract

Psychological harms from interactions with large language models cannot be monitored directly because they may emerge over repeated interactions and depend strongly on subjective individual user context. We propose measuring risk, not harm: detecting observable AI behaviors that constitute risk factors for psychological harm at the AI response level and aggregating these signals to support longitudinal monitoring. We operationalize eight such behaviors and construct a synthetic multi-label corpus of 47,062 AI responses and 373,873 response-factor labels, with targeted human annotations for validation and evaluation. We then compare trained embedding-based classifiers, direct LLM judging, and a classifier--judge cascade. All detectors are model-agnostic and operate only on LLM outputs, requiring no access to internal model states. We apply these detectors to several out-of-distribution human-AI conversation data sets from prior research. On held-out human-labeled data, the best cascade achieves macro precision of 0.86, recall of 0.76, and F1 of 0.79. On WildChat, it reduces LLM-judge calls by 97.1\% relative to direct judging. On DelusionEval, risk-factor detections cluster within conversations. Together, our results show that response-level behavioral risk-factor detection can provide a scalable measurement layer for longitudinal psychological-safety monitoring without conflating observable AI behavior with latent user harm.
BibTeX
@misc{akpinar2026measuring,
  author = {Nil-Jana Akpinar and Eugenia Kim and Natalie Isak and Amanda Minnich and Victoria Westerhoff and Matthew Dressman},
  title = {{Measuring Risk, Not Harm: Detecting Psychological Harm Risk Factors in LLM Conversations}},
  year = {2026},
  eprint = {2611.06806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06806},
}
Figure