OpenAI released MentalHealthBench, an open benchmark that measures how AI systems respond in realistic mental health conversations, on Sept. 23.
The company built the benchmark with more than 80 licensed psychologists and psychiatrists from 22 countries. The cohort speaks 19 languages and represents nearly 20 mental health subspecialties. The benchmark scores model behaviors including safety, context-seeking, preservation of user agency and actionable guidance.
OpenAI said most earlier evaluations in this area focused on emergencies and relied on broad, predefined criteria. MentalHealthBench covers nonacute, high-acuity and emergency scenarios involving adults, teens ages 13-17, caregivers and clinicians across multiple languages and regions. OpenAI used privacy-preserving techniques to generate synthetic conversations that reflect real-world usage patterns.
At least three experts reviewed each conversation and wrote rubric criteria weighted from -10 to +10. OpenAI kept only criteria that at least two experts agreed on and a third did not contradict. An automated grader, GPT-5.6 Sol, scores model responses against those criteria.
A separate analysis of 44 adults who had used AI for mental health or emotional support found that users placed more weight on practical next steps and tone. Participants reviewed only nonacute conversations. Experts placed more weight on gathering context and interpreting ambiguous situations.
“Mental health exists on a continuum, from flourishing to everyday stress to acute crisis,” said Arthur Evans, PhD, CEO of the American Psychological Association. Dr. Evans said AI systems operating across that range need grounding in both clinical science and lived experience.
OpenAI, whose ChatGPT reaches more than 1 billion people each week, released the benchmark openly so outside researchers can examine its methods and run their own evaluations.
At the Becker's Fall Behavioral Health Summit, taking place November 4–5 in Chicago, behavioral health leaders and executives will explore strategies for expanding access to care, integrating services, addressing workforce challenges and leveraging innovation to improve outcomes across the behavioral health continuum. Apply for complimentary registration now.
