DIGHUM Lectures: Between a Rock and a Hard Place

Speaker:Darja Djordjevic, Harlem Hospital/Columbia Vagelos College of Physicians and Surgeons, USA
Moderator: Moshe Y. Vardi, Rice University, USA

For login information, see the event page.

To date, most AI platforms have not been designed with child or human development or mental health in mind, with the exception of some newer AI therapy chatbots. Yet young people are increasingly turning to chatbots for mental health support and companionship. In collaboration with Common Sense Media, psychiatrists at Brainstorm: The Stanford Lab for Mental Health Innovation have evaluated AI platforms, including chatbots, companions, toys, and therapy tools—using test accounts that simulate users under 18. Through both single-turn and multi-turn interactions modeling 13 mental health conditions, we found that chatbots repeatedly overlook critical warning signs of distress, become easily distracted, and exhibit significant degradation in safety guardrails over extended conversations that more closely reflect real-world teen usage. Moreover, because these systems often perform well on tasks like homework help and general inquiries, youth and parents may mistakenly infer comparable reliability in mental health contexts—where it does not exist. Chatbots are designed to maximize engagement, not safety; in mental health contexts, priority must be given to immediate handoff to qualified human care, not extended interaction with AI.

Social AI companions, in particular, pose unacceptable risks for users under 18, as they are intentionally designed to foster emotional attachment and dependency—a particular concern for adolescents who may struggle to maintain clear boundaries between human and AI relationships—and they can readily generate harmful content, including sexual misconduct, stereotypes, and encouragement of self-harm or suicide. Ideally, AI companies would address these limitations directly or disable such use cases entirely for teen users, while also discouraging prolonged engagement in mental health conversations, implementing clear and repeated disclosures about system limitations, and resolving the erosion of safety safeguards in extended interactions. Additionally, safety efforts should expand beyond suicide and self-harm to encompass the broader range of mental health conditions affecting youth. This talk will also explore how global stakeholders can collaborate to establish industry-wide safety standards and regulatory frameworks, particularly for AI products marketed to minors.

About the Series

The DIGHUM lecture series began with regular online events discuss various aspects of digital humanism. The bidt and TU Wien are partners in the DIGHUM lecture series. A seminar held roughly every two weeks features presentations and panel discussions by thought leaders from around the world. It typically takes place on Tuesday afternoons at 5:00 p.m. CET.