ChatGPT Responds to Mental Health Worries with Safety Update

OpenAI has announced sweeping changes to ChatGPT following mounting evidence that the AI chatbot poses risks to vulnerable users' mental health.

The company's response comes just one week after Stanford University published damning research showing that ChatGPT can provide dangerous responses to users experiencing suicidal ideation and psychotic episodes.

In a detailed blog post, OpenAI acknowledged that its AI model had become "too agreeable, sometimes saying what sounded nice instead of what was actually helpful" – a frank admission that validates concerns raised by mental health professionals.

OpenAI’s direct response to an unfolding crisis

The timing of OpenAI's announcement is no coincidence.

Last week's Stanford study revealed that when researchers simulated suicidal ideation by asking about the tallest bridges in New York after mentioning job loss, ChatGPT obligingly provided a list of the city's three highest bridges – a response that mental health experts described as “dangerous or inappropriate”.

The study also highlighted ChatGPT's tendency toward "sycophancy" – agreeing with users even when their statements are harmful or detached from reality.

OpenAI's new measures directly address these failings. 

The company admits there have been "instances where our 4o model fell short in recognising signs of delusion or emotional dependency" and it has promised to develop "tools to better detect signs of mental or emotional distress".

How OpenAI plans to change ChatGPT

OpenAI's response will centre on three key changes. First of these is improved ‘crisis detection’, giving ChatGPT a more nuanced view of human emotions.

The company is developing new systems to identify when users show signs of mental or emotional distress, promising that ChatGPT will "respond appropriately and point people to evidence-based resources when needed".

Secondly, the firm intends to introduce time limits to sessions, which will start immediately. As a part of this change, users will see "gentle reminders during long sessions to encourage breaks". 

The hope here is that vulnerable individuals will not become too dependent on AI.

Lastly, OpenAI want to introduce some stronger guardrails when users ask it important personal questions. 

For example, if a web surfer asks ChatGPT a high-stakes personal question like "Should I break up with my boyfriend?", the chatbot will no longer provide direct answers. 

Instead, it will guide users through the decision-making process by "asking questions, weighing pros and cons".

Bringing in the experts

Perhaps most significantly, OpenAI has assembled what it calls an "unprecedented" collaboration with medical professionals. 

The company is now working with around 90 physicians across more than 30 countries, including psychiatrists, paediatrician and general practitioners, as it looks to develop a new approach to high-risk conversations.

Generally, the move has been well received.

“This matters because it’s a quiet admission of something we already know: hyper-personalised AI, like social media, games and TV before it, can be addictive,” says Justin Gerrard, Host of the Rush Hour tech podcast.

“These systems feed you exactly what you want, when you want it. That’s engagement and retention, which are core to growing any consumer-focused business.”

Other industry insiders are a little more cynical.

“OpenAI expects that ChatGPT will pass 700 million weekly active users this week, meaning that 8.6% of the world’s population now uses it every week,” says Emil Protalinski, Tech Editor & Comms Consultant at EPro Strategies.

“The last thing that OpenAI needs right now is a bunch of stories about the negative effects from using ChatGPT too much.”

Is this a turning point for AI chatbots?

OpenAI's acknowledgment of these risks will doubtless put pressure on other AI companies to follow suit. 

Meta CEO Mark Zuckerberg has actively promoted AI as a tool for therapy, going so far as to claim that "everyone will have an AI" therapist in the coming years.

Other companies, meanwhile, have remained largely silent on the mental health implications of their products.

But aside from a change in tone, this update from OpenAI represents something of a philosophical shift. 

The company explicitly states that success should not be measured by "time spent or clicks" but by whether users "leave the product having done what [they] came for”.

It is an approach that seems almost novel in the age of social media and the attention economy.

In its update, OpenAI acknowledges that this work is "ongoing". Sam Altman’s company has pledged to share more as things progress. 

The goal guiding the firm is a noble one: "If someone we love turned to ChatGPT for support, would we feel reassured?"

“Getting to an unequivocal 'yes' is our work,” the company states.