Mindbench.ai: an actionable platform to evaluate the profile and performance of large language models in a mental healthcare context

Bridget Dwyer, Matthew Flathers, Akane Sano, Allison G. Dempsey, Andrea Cipriani, Asim H. Gazi, Bryce Hill, Carla Gorban, Carolyn I. Rodríguez, C.f. Stromeyer, Darlene King, Eden Rozenblit, Gillian Strudwick, Jake Linardon, Jiaee Cheong, Joseph Firth, Julian Herpertz, Julian Schwarz, Khai N. Truong, Margaret R. Emerson, Martin P. Paulus, Michelle A. Patriquin, Yining Hua, Soumya Choudhary, Steven Siddals, Laura Ospina‐Pinillos, Jason Bantjes, Steven Scheuller, Xuhai Xu, Ken Duckworth, Daniel H. Gillison, Michael B. Wood, John Torous

Abstract

Individuals are increasingly utilizing large language model (LLM)-based tools for mental health guidance and crisis support in place of human experts. While AI technology has great potential to improve health outcomes, insufficient empirical evidence exists to suggest that AI technology can be deployed as a clinical replacement; thus, there is an urgent need to assess and regulate such tools. Regulatory efforts have been made and multiple evaluation frameworks have been proposed, however, field-wide assessment metrics have yet to be formally integrated. In this paper, we introduce a comprehensive online platform that aggregates evaluation approaches and serves as a dynamic online resource to simplify LLM and LLM-based tool assessment: MindBench.ai. At its core, MindBench.ai is designed to provide easily accessible/interpretable information for diverse stakeholders (patients, clinicians, developers, regulators, etc.). To create MindBench.ai, we built off our work developing MINDapps.org to support informed decision-making around smartphone app use for mental health, and expanded the technical MINDapps.org framework to encompass novel large language model (LLM) functionalities through benchmarking approaches. The MindBench.ai platform is designed as a partnership with the National Alliance on Mental Illness (NAMI) to provide assessment tools that systematically evaluate LLMs and LLM-based tools with objective and transparent criteria from a healthcare standpoint, assessing both profile (i.e. technical features, privacy protections, and conversational style) and performance characteristics (i.e. clinical reasoning skills). With infrastructure designed to scale through community and expert contributions, along with adapting to technological advances, this platform establishes a critical foundation for the dynamic, empirical evaluation of LLM-based mental health tools—transforming assessment into a living, continuously evolving resource rather than a static snapshot.

Introduction: The Need for Rigorous AI Assessment

As Large Language Models (LLMs) like ChatGPT and Claude become more sophisticated, they are increasingly being used by the public for emotional support and crisis guidance. However, these tools often lack the empirical evidence required for safe clinical deployment. Existing medical benchmarks, such as MedQA, are frequently too broad, with psychiatry representing only a small fraction of the testing data.

Mental healthcare presents unique challenges: it requires nuanced conversational dynamics, strict privacy protections, and a deep understanding of the risks associated with "AI Psychosis" or harmful over-reliance. Currently, there is no centralized, standardized method to audit how different models handle sensitive clinical scenarios or to compare their underlying technical frameworks. MindBench.ai addresses this gap by providing a transparent, "living" resource for clinicians, researchers, and developers.

The MindBench.ai Framework

The authors developed a comprehensive two-pronged evaluation framework to assess both the identity and the capability of LLMs in a mental health context.

MindBench.ai Evaluation Framework

1. Profile Evaluation

This branch assesses the "who" and "how" of the model:

  • Technical Profile: Leveraging the MINDapps.org framework, the platform uses 107 specific questions to audit data retention policies, API reliability, and security certifications.
  • Conversational Dynamics: Recognizing that users often anthropomorphize AI, MindBench.ai maps model responses to validated personality scales (e.g., Big Five, HEXACO, MBTI). This helps identify "default" interaction styles that might be overly sycophantic or inappropriately clinical.

2. Performance Evaluation

This branch assesses the "what" of the model's output:

  • Clinical Benchmarking: Moving beyond binary right/wrong scores, the platform utilizes the Suicide Intervention Response Inventory (SIRI-2). This uses numeric expert ratings to capture the "graded" appropriateness of a response in high-stakes crisis situations.
  • Reasoning Analysis: The platform implements Chain-of-Thought (CoT) extraction to force models to articulate their step-by-step logic. It also employs adversarial testing to see where a model’s logic breaks down when faced with anchoring biases or information gaps.

Data Sources and Methodology

MindBench.ai aggregates several high-quality data sources for its evaluations:

  • MINDapps.org: Provides the foundational 105-dimension framework for profiling.
  • SIRI-2: A validated clinical scale for crisis-response evaluation.
  • New Clinical Suite: Developed in collaboration with psychiatry residents at Beth Israel Deaconess Medical Center (BIDMC), this includes 75 new clinical case benchmarks covering psychopharmacology, perinatal mental health, and psychiatric diagnosis.
  • Personality Inventories: Standardized items from the International Personality Item Pool (IPIP) are used to evaluate LLM "personalities."

Key Insights: Personality as Safety

A central insight of this work is that a model's "personality" is a safety metric. For instance, a model that scores too high on "agreeableness" might inadvertently validate a patient's harmful delusions or self-destructive plans. By using expert-rated Likert scales (via SIRI-2), MindBench.ai captures the clinical nuance that traditional machine learning benchmarks miss.

Furthermore, the platform is designed to be a "living" benchmark. As models update (e.g., from GPT-4 to GPT-4o), the platform can automatically re-run evaluations, ensuring that stakeholders have access to current, actionable data in a rapidly evolving field.

Platform Results and Implementation

MindBench.ai is currently live, featuring a leaderboard that compares major models such as GPT-4o, Claude 3.5, and Gemini 2.5 Pro.

Model Personality Comparison Interface

The platform provides granular leaderboards rather than a single "IQ score." This allows users to see, for example, if one model excels at crisis intervention while another is better suited for psychopharmacology. The interface allows for side-by-side technical and performance comparisons of up to three models, complete with verification sources and timestamps. By highlighting areas where even human clinicians disagree, the platform provides a realistic baseline for AI performance.

Limitations and Future Directions

While MindBench.ai represents a significant step forward, several limitations remain:

  • Cultural Bias: Current benchmarks are primarily developed by Western, English-speaking clinicians, which may limit global applicability.
  • Prompt Sensitivity: A user's specific prompt can override default model behaviors. Future work will include representative "usage patterns" in the evaluation library.
  • Long-term Outcomes: The platform currently measures snapshot performance; evaluating the impact of sustained use on patient well-being is a priority for future research.
  • Community Expansion: The authors are calling for multilingual benchmarks and cultural adaptation tracking to make the platform more globally representative.

Summary updated

Abstract

Individuals are increasingly utilizing large language model (LLM)-based tools for mental health guidance and crisis support in place of human experts. While AI technology has great potential to improve health outcomes, insufficient empirical evidence exists to suggest that AI technology can be deployed as a clinical replacement; thus, there is an urgent need to assess and regulate such tools. Regulatory efforts have been made and multiple evaluation frameworks have been proposed, however,field-wide assessment metrics have yet to be formally integrated. In this paper, we introduce a comprehensive online platform that aggregates evaluation approaches and serves as a dynamic online resource to simplify LLM and LLM-based tool assessment: MindBench.ai. At its core, MindBench.ai is designed to provide easily accessible/interpretable information for diverse stakeholders (patients, clinicians, developers, regulators, etc.). To create MindBench.ai, we built off our work developing MINDapps.org to support informed decision-making around smartphone app use for mental health, and expanded the technical MINDapps.org framework to encompass novel large language model (LLM) functionalities through benchmarking approaches. The MindBench.ai platform is designed as a partnership with the National Alliance on Mental Illness (NAMI) to provide assessment tools that systematically evaluate LLMs and LLM-based tools with objective and transparent criteria from a healthcare standpoint, assessing both profile (i.e. technical features, privacy protections, and conversational style) and performance characteristics (i.e. clinical reasoning skills). With infrastructure designed to scale through community and expert contributions, along with adapting to technological advances, this platform establishes a critical foundation for the dynamic, empirical evaluation of LLM-based mental health tools-transforming assessment into a living, continuously evolving resource rather than a static snapshot.
BibTeX
@inproceedings{dwyer2025mindbench,
  author = {Bridget Dwyer and Matthew Flathers and Akane Sano and Allison G. Dempsey and Andrea Cipriani and Asim H. Gazi and Bryce Hill and Carla Gorban and Carolyn I. Rodríguez and C.F. Stromeyer and Darlene King and Eden Rozenblit and Gillian Strudwick and Jake Linardon and Jiaee Cheong and Joseph Firth and Julian Herpertz and Julian Schwarz and Khai N. Truong and Margaret R. Emerson and Martin P. Paulus and Michelle A. Patriquin and Yining Hua and Soumya Choudhary and Steven Siddals and Laura Ospina‐Pinillos and Jason Bantjes and Steven Scheuller and Xuhai Xu and Ken Duckworth and Daniel H. Gillison and Michael B. Wood and John Torous},
  title = {{Mindbench.ai: an actionable platform to evaluate the profile and performance of large language models in a mental healthcare context}},
  booktitle = {NPP—Digital},
  year = {2025},
  doi = {10.1038/s44277-025-00049-6},
}
Figure