OpenAI Launches MentalHealthBench to Evaluate AI Responses in Mental Health Scenarios

OpenAI has released MentalHealthBench, an open benchmark designed to evaluate how AI systems respond in real-world mental health conversations. The dataset includes 1,215 synthetic dialogues spanning everyday mental health topics to urgent crisis situations.

The benchmark was developed with input from more than 80 licensed psychologists and psychiatrists across 22 countries and regions, who used 19 languages and covered nearly 20 specialty areas. The full dataset contains 5,262 expert-written scoring rubrics, with each dialogue reviewed by at least three experts. Non-acute dialogues account for 53.5% of the dataset, high-risk dialogues 18.2%, and urgent dialogues 28.3%.

OpenAI said the synthetic dialogues were generated using privacy-preserving techniques described in a companion paper, with a method similar to its Clio system, aiming to reflect real-world usage patterns of ChatGPT. In evaluations, an automated scorer based on GPT-5.6 Sol generated four responses per task, and scores were presented as task-clipped rubric scores. GPT-6 Astra achieved the highest score at 57.3%, followed by GPT-6 Sol at 53.9% and Claude Opus 5.5 at 52.4%.

With the public release, OpenAI said it hopes other researchers can scrutinize the methodology, run independent evaluations, and build on the work. The company also reiterated that ChatGPT is not a substitute for psychotherapy or professional medical care.