On-Call

Ultimately, the [Trusted Contact] feature puts the responsibility back on the user, adds [Holly] Wilcox[, director of the Center for Suicide Prevention at the Johns Hopkins Bloomberg School of Public Health]. She would like to see companies like OpenAI be more proactive in partnering with existing crisis resources like 988.

"So maybe having a human jump into the conversation at some point who's trained and knows the tools of what to do for suicide prevention," she says. "And have them jump in and say, 'You know, what's going on? Can I help?"
She told no one about her agony except ChatGPT. What her death reveals about AI risks

I've heard this suggestion made before, when it comes to generative automation chatbots and people in mental heath crises or otherwise at risk. But so far, I haven't heard the question of "who would these humans be" addressed. And that's important. Not to mention how this would all work.

If "a human" receives a ping from ChatGPT or Copilot or whatever that a person it is communicating with is showing signs of being "at risk," what should come along with that? Are they dumped into the conversation cold, and have to then sort out who the person is, and what's going on, in real time? Are they given information that the user has already given the chatbot? If so, how much? And what happens if the person communicating with the chatbot doesn't want to talk to "a human?" Is "a human" allowed to force the issue, and, if the user is signed into the service that summoned them, force the service to not re-open communications with the user unless "a human" gives the all-clear? What can, or should, companies program into their user interfaces to detect people attempting to work around safeguards? What will be the tolerance for false positives, as well as false negatives? For the idea to be viable, these questions will have to be answered.

To be sure, "maybe having a human jump into the conversation at some point" isn't a bad suggestion on the face of it, but it's a very simple suggestion on a way to solve a very complicated problem, and the sorts of news stories in which this pops up don't often have the space to go into the depth required. The NPR story that I quote from to lead off this post isn't about how to solve the problems associated with people turning to LLM-powered chatbots for mental health advice; it's the story of an attractive woman who killed herself, and her family's heartbreak, driven by her journalist mother. This is a human-interest story not a technology story. And I suspect that it feeds neatly into the overall suspicion of LLMs, and the companies that make them, on the part of the NPR audience.

And there's nothing wrong with human interest stories. There's nothing wrong with a sympathetic portrayal of a woman, and her parents, designed to bring attention to a tragic situation and a potential risk factor of an emergent technology. But there does need to be some follow-up. "Maybe having a human jump into the conversation at some point," requires, at a minimum, a number of people who are going to be on-call, such that there are always people available, so that there is going to be "a human" available, whenever a chatbot detects some level of a person being at risk. That, in and of itself, is going to have measurable costs. In 2023, the wait time for the 988 Suicide and Crisis Hotline was less than a minute. Understanding what it will take to keep the wait time low will be important. Not to mention the idea that there are likely some number of people who are turning to ChatGPT or Copilot or a different service specifically because they mistrust the 988 service.

But half of LGBTQ+ people in a recent survey by Pew Charitable Trusts said they worried a call to 988 could end in an unwanted interaction with police or a forced hospitalization.
Mental health crisis line 988 fields significantly more calls, 12 times as many texts in first year

Regardless of how realistic a concern that is, given an Administration (and 988 is a federal program) that has show itself to be hostile to the LGBT community, it's not necessarily an irrational one (and even if it was, we are talking about a mental heath service here). And, speaking of the current Administration, it also stands to reason that immigrants could be leery about using 988.

So there's a concern, worth taking seriously, that part of the reason why people turn to generative automation for mental health advice and guidance is that the resources with actual people behind them carry some sort of real or perceived risk for them.

I don't think that any of this is insurmountable. But it does require addressing, at least if a program is going to be successful. So I'd like to see a news story, or a podcast, perhaps, that actually looks at what it would take to build out this idea.