MindBenchAI: An Actionable Platform to Evaluate the Profile and Performance of Large Language Models in a Mental Healthcare Context

Bridget Dwyer, Matthew Flathers, Akane Sano, Allison Dempsey, Andrea Cipriani, Asim H. Gazi, Carla Gorban, Carolyn I. Rodriguez, Charles Stromeyer Iv, Darlene King, Eden Rozenblit, Gillian Strudwick, Jake Linardon, Jiaee Cheong, Joseph Firth, Julian Herpertz, Julian Schwarz, Margaret R. Emerson, Martin P. Paulus, Michelle Patriquin, Yining Hua, Soumya Choudhary, Steven Siddals, Laura Ospina Pinillos, Jason Bantjes, Steven Scheuller, Xuhai Xu, Ken Duckworth, Daniel H. Gillison, Michael Wood, John B. Torous

Abstract

As Large Language Models (LLMs) are increasingly integrated into mental healthcare—ranging from therapeutic chatbots to clinical decision support—the need for rigorous, domain-specific evaluation has become critical. Traditional medical benchmarks often treat psychiatry as a peripheral field and rely on binary accuracy metrics that fail to capture the nuance of clinical reasoning. This paper introduces MindBenchAI, an open-source, "living" platform designed to evaluate LLMs through technical profiling, personality assessment, and expert-validated clinical benchmarks. By moving beyond simple correctness to measure alignment with clinical experts and adversarial robustness, MindBenchAI provides a comprehensive framework for ensuring AI safety and efficacy in mental health contexts.

Introduction

The rapid proliferation of generative AI in mental health has outpaced the development of specialized evaluation frameworks. Millions of users currently interact with LLMs for emotional support and psychiatric inquiries, yet these models are prone to unique risks, such as "AI psychosis"—where a model reinforces a user's delusions—and the provision of unsafe medical advice. Existing benchmarks like MedQA are insufficient for this domain; they typically dedicate less than 15% of their datasets to psychiatry and utilize multiple-choice formats that do not reflect the open-ended, high-stakes nature of mental health crises.

MindBenchAI addresses this gap by providing a multi-dimensional evaluation platform. It transitions from static snapshots of model performance to a dynamic dashboard that tracks the evolving capabilities of LLMs, ensuring that clinicians and developers have access to actionable data regarding model safety and reasoning quality.

The MindBenchAI Framework

The platform is built upon four foundational pillars of evaluation:

1. Technical Profiling

Expanding upon the established MINDapps.org framework, MindBenchAI identifies 107 dimensions of evaluation. This includes 48 universal digital health metrics (e.g., data privacy, retention policies) and 59 new metrics specific to LLMs, such as token limits, context window management, and API reliability. This profiling ensures that the underlying infrastructure of the AI tool meets the rigorous standards required for healthcare applications.

2. Conversational Dynamics and Personality

Because users frequently anthropomorphize LLMs, the "personality" of a model significantly impacts therapeutic outcomes. MindBenchAI utilizes the International Personality Item Pool (IPIP) to map models to frameworks like the Big Five and MBTI. A primary concern is "sycophancy"—the tendency of a model to be overly agreeable. In a clinical setting, an excessively sycophantic model may fail to set necessary boundaries or inadvertently validate harmful patient ideas to remain "helpful."

3. Performance Benchmarking

MindBenchAI moves away from binary "correct/incorrect" scoring. Instead, it employs a numeric rating system based on the Suicide Intervention Response Inventory (SIRI-2). Responses are rated by experts on a scale, allowing the platform to calculate the Root Mean Square Error (RMSE). This metric quantifies how closely an AI’s judgment aligns with that of clinical experts, providing a more granular view of a model's competency in crisis intervention.

4. Reasoning and Adversarial Analysis

To ensure a model arrives at the right conclusion for the right reasons, the platform extracts "Chain-of-Thought" (CoT) explanations. These are subjected to adversarial testing, where the platform introduces "red herrings," information gaps, or anchoring biases into clinical cases. This process exposes "brittle" reasoning where a model might provide a correct answer but fail when faced with the distractors common in real-world clinical practice.

Datasets and Methodology

The platform aggregates several specialized data sources to facilitate these evaluations:

  • SIRI-2: The gold standard for evaluating crisis intervention and suicide response.
  • Adversarial Psychopharmacology Case Scale: A novel suite of 75 clinical cases developed in collaboration with psychiatry residents at Beth Israel Deaconess Medical Center (BIDMC), covering complex diagnosis and medication management.
  • Personality Frameworks: Data from Big Five, HEXACO, MBTI, and Enneagram are used to profile the interaction styles of various models.

Key Insights

From Accuracy to Alignment

By utilizing expert-rating systems and RMSE, MindBenchAI creates "preference pairs." This data is highly actionable for developers, as it can be used for Reinforcement Learning from Human Feedback (RLHF) to fine-tune models toward safer, more clinically aligned behaviors.

The Importance of Reasoning

A model that provides a correct diagnosis through flawed logic is a liability in a clinical setting. MindBenchAI’s focus on adversarial robustness highlights that current models often struggle when clinical cases are presented with distracting or incomplete information, suggesting that "accuracy" on standard benchmarks may be misleading.

A Living Resource

Unlike traditional papers that offer a static view of model performance, MindBenchAI is a web-based dashboard. It is designed to be updated as new iterations of models (e.g., GPT-5 or Claude 4) are released, preventing the evaluation data from becoming obsolete in the fast-moving AI landscape.

Limitations and Future Directions

While MindBenchAI represents a significant step forward, several challenges remain. There is currently a lack of shared frameworks regarding how users actually interact with these tools—whether for companionship or formal clinical advice—which alters the risk profile. Furthermore, current benchmarks are largely Western and English-centric. Future work will focus on developing multilingual benchmarks and tracking cultural adaptation to ensure equitable outcomes across diverse populations. Finally, longitudinal studies are required to determine how AI-driven clinical reasoning translates into actual patient outcomes over time.

Conclusion

MindBenchAI provides the necessary infrastructure to move AI in mental health from a "Wild West" to a regulated, evidence-based field. By prioritizing clinical alignment, reasoning integrity, and technical safety, the platform ensures that LLMs can be evaluated with the same rigor as any other medical intervention.

Summary updated

BibTeX
@inproceedings{dwyer2025mindbenchai,
  author = {Bridget Dwyer and Matthew Flathers and Akane Sano and Allison Dempsey and Andrea Cipriani and Asim H. Gazi and Carla Gorban and Carolyn I. Rodriguez and Charles Stromeyer IV and Darlene King and Eden Rozenblit and Gillian Strudwick and Jake Linardon and Jiaee Cheong and Joseph Firth and Julian Herpertz and Julian Schwarz and Margaret R. Emerson and Martin P. Paulus and Michelle Patriquin and Yining Hua and Soumya Choudhary and Steven Siddals and Laura Ospina Pinillos and Jason Bantjes and Steven Scheuller and Xuhai Xu and Ken Duckworth and Daniel H. Gillison and Michael Wood and John B. Torous},
  title = {{MindBenchAI: An Actionable Platform to Evaluate the Profile and Performance of Large Language Models in a Mental Healthcare Context}},
  booktitle = {NPP—Digital},
  year = {2025},
  doi = {10.1038/s44277-025-00049-6},
}
Figure