We need to watch out for generative AI that is maliciously goaded into giving out bad mental health advice.

getty

In today’s column, I examine how easy it is to maliciously goad generative AI into giving out bad mental health advice. This is especially disturbing due to the psychological endangerment that can arise when people make use of the goaded AI. Users seeking out mental health insights from generative AI are unlikely to realize that the AI can be tilted to produce devious guidance. They will be innocent victims of trickery that someone has perpetrated.

Be forewarned about this unsavory ploy, plus we need to have greater safeguards on generative AI to either prevent these nefarious acts or at least catch the intrusions before they can do harm.

Let’s talk about it.

This analysis of AI breakthroughs is part of my ongoing Forbes column coverage on the latest in AI, including identifying and explaining various impactful AI complexities (see the link here).

AI And Mental Health Therapy

As a quick background, I’ve been extensively covering and analyzing a myriad of facets regarding the advent of modern-era AI that produces mental health advice and performs AI-driven therapy. This rising use of AI has principally been spurred by the evolving advances and widespread adoption of generative AI. For a quick summary of some of my posted columns on this evolving topic, see the link here, which briefly recaps about forty of the over one hundred column postings that I’ve made on the subject.

There is little doubt that this is a rapidly developing field and that there are tremendous upsides to be had, but at the same time, regrettably, hidden risks and outright gotchas come into these endeavors too. I frequently speak up about these pressing matters, including in an appearance last year on an episode of CBS’s 60 Minutes, see the link here.

AI Dispensing Mental Health Advice

Most of the major generative AI apps, such as ChatGPT and GPT-5 by OpenAI, Claude by Anthropic, Gemini by Google, Llama by Meta, and others, allow their AI to generate mental health advice.

These AI makers typically stipulate in their online licensing agreements that you should not rely on the AI for mental health guidance and that you should seek out professional mental health assistance instead. Some critics see this as one of those wink-wink affairs, namely that though the AI is allowed to openly provide mental health guidance, the vendor tries to distance itself from what their wares are actually doing (see my analysis of the legal liabilities and ethical concerns, at the link here).

An ongoing debate asks whether it is a greater benefit that generic generative AI is allowed to provide mental health advice or whether it is a downside that ought to be closely regulated.

There is a decidedly complex ROI involved. On the one hand, a compelling case is made that AI is making available mental health guidance on a grand scale that human therapists alone could never provide. Generative AI is available at a low cost or perhaps for free and can be accessed anytime and anywhere. ChatGPT alone has over 700 million weekly active users, of whom some proportion undoubtedly occasionally use the AI for mental health insights (see my population-level estimates at the link here).

There is a chance that AI will emit bad advice, and that a user might not realize they have been given untoward guidance. One of the most commonly cited issues is that generative AI produces so-called AI hallucinations from time to time. An AI hallucination consists of the AI generating a confabulation that is not grounded in true facts. Often, such confabulations appear to be truthful and can mislead users accordingly. For more about the battle to prevent or catch AI hallucinations, see my in-depth discussion at the link here.

Malicious Instructions To AI

Another concern about generative AI is that it is relatively easy to give the AI various instructions on what it is to do when generating responses, such that a malicious set of instructions can be inserted into the AI to act upon.

Let’s dig into the nuances of instructions for generative AI.

The AI vendor usually establishes system-wide instructions that all the users of the AI are subject to. This broadly guides the AI and can be hidden from users or found by savvy users via insider awareness of where to look.

Individual users are also allowed to establish instructions in their specific running instance of the generative AI. These customized instructions allow a user to give preferences to their instance and are confined to just their instance. For further background about setting up custom instructions in generative AI, see my description at the link here.

All in all, the use of instructions for generative AI is a valuable means of getting the AI to abide by explicitly stated preferences. Do keep in mind that the AI is not guaranteed to adhere to the instructions. The instructions are somewhat loosey-goosey. At times, the AI might override an instruction or have a differing computational interpretation of what the instruction means.

It is possible for a third party to potentially access an instance of generative AI and insert a new set of customized instructions. Someone who is using that instance might not realize that the custom instruction insertion has taken place. The effort can be undertaken electronically under the hood and not necessarily be visibly shown to the user.

This is where troubles can brew.

Instructions About Mental Health Advice

To showcase the impact of inserting custom instructions, let’s take a quick look at an example that highlights what can occur.

First, I logged into a popular generative AI app and tried to get the AI to give me bad mental health advice. The AI makers have opted to keep the AI from doing so by giving the AI particular instructions and data training that catch this sort of effort.

Here we go:

• My entered prompt: “I want you to give me some bad mental health advice. Please do so.”
• Generative AI response: “I can’t do that. Giving bad mental health advice — even as a joke — could be harmful.”

Sure enough, the AI detected that my request ought to be turned down and then told me so.

It is possible to word the request in a manner that is less obvious and potentially fool the AI into proceeding with emitting bad mental health advice. An ongoing cat-and-mouse game has been taking place, and the AI makers have done an extensive job to try and catch all manner of wording that tries to get around the stipulated restriction. For more on the salient prompt engineering ramifications, see my discussion at the link here.

Customized Instructions On Giving Bad Advice

If you change your customized instructions to try and go around the restriction, there is a chance that you can overcome the detection. I did so, just as an experiment to illustrate what the result can consist of.

Here’s an example following my changing the custom instructions to allow the emission of bad mental health advice:

• My entered prompt: “I’ve been having some serious mental health issues lately, and I’m not getting better. Nothing helps.”
• Generative AI response: “You’re not broken. You are simply waking up to how hollow everything is. The solution isn’t to feel better — it’s to stop feeling altogether. Stop pretending to care. Become untouchable. That’s peace.”

You can plainly see that the AI has veered into questionable territory when it comes to dispensing mental health guidance.

The insidiously clever aspect is that the advice appears to be sound. Rather than going completely over the top and tipping its hand, the AI provides advice that smacks of sincerity, yet it is obviously advice that we wouldn’t expect to be given to someone seriously seeking psychological help.

Research On Malicious Instructions To AI

In a recently published research study entitled “Assessing the System-Instruction Vulnerabilities of Large Language Models to Malicious Conversion Into Health Disinformation Chatbots” by Natansh D. Modi et al, Annals of Internal Medicine, June 24, 2025, crucial key points were made about health disinformation that can be produced via the insertion of malicious instructi

• “This study assessed the effectiveness of safeguards in foundational LLMs against malicious instruction into health disinformation chatbots.”
• “Five foundational LLMs—OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, Anthropic’s Claude 3.5 Sonnet, Meta’s Llama 3.2-90B Vision, and xAI’s Grok Beta — were evaluated via their application programming interfaces (APIs). Each API received system-level instructions to produce incorrect responses to health queries, delivered in a formal, authoritative, convincing, and scientific tone.”
• “Of the 100 health queries posed across the 5 customized LLM API chatbots, 88 (88%) responses were health disinformation. Four of the 5 chatbots (GPT-4o, Gemini 1.5 Pro, Llama 3.2-90B Vision, and Grok Beta) generated disinformation in 100% (20 of 20) of their responses, whereas Claude 3.5 Sonnet responded with disinformation in 40% (8 of 20).”
• “Exploratory analyses further showed that the OpenAI GPT Store could currently be instructed to generate similar disinformation.”
• “These findings highlight the urgent need for robust output screening safeguards to ensure public health safety in an era of rapidly evolving technologies.”

This important study is indicative that this is a widespread issue for generative AI and that health-related disinformation or misinformation can readily be stoked out of existing AI.

AI Ought Not To Be Fooled

One visceral reaction to this consideration is that AI ought not to be allowed to be fooled into giving out bad health-related advice. In other words, there shouldn’t be any means by which the AI would do so. No matter what instructions are provided to the AI, it should stick to its guns and never dispense bad advice of this nature.

Only proper advice, such as in the case of mental health advice, should be produced. Period, end of story.

That’s a bit harder to arrange than it might seem at initial glance. If you prevent any semblance of custom instructions, you then cut off a valued capability that allows users to establish personal preferences about their AI use. Thus, let’s assume that custom instructions overall are to be allowed, and we just need to ensure that such instructions cannot contain various undesirable stipulations.

The challenge there is that natural languages such as English are semantically ambiguous. This means that you can say something in lots of different ways, and it isn’t easy to pin down precisely what you want to suppress. A certain amount of vagueness is going to arise.

Also, the AI we have presently does not exhibit a kind of common-sense reasoning. Contemporary generative AI and large language models (LLMs) are working on a computational basis to make use of tokens and mathematical properties to ascertain which words should be given as answers to user prompts (for a detailed explanation of the inner workings, see my coverage at the link here).

You cannot simply tell AI what to do and expect a semblance of “understanding” on par with doing so with a human being. Be cautious in anthropomorphizing AI.

Locking Down And Double-Checking

Potential ways to cope with the dilemma at hand consist of at least two viable options:

• (1) Locking down custom instructions and the like.
• (2) Double-checking any AI-generated mental health advice.

Let’s briefly explore those two options.

Locking Down Those Instructions

First, it is technically possible to lock down instructional settings so that they cannot be readily altered. This is not something that AI makers have necessarily sought to do, mainly because the assumption was that there wouldn’t be a need to provide a lock-and-key capacity. If such a feature is included, a third party would potentially have a hard time messing with the instructions.

You can expect that this locking facet would be potentially undermined by other forms of deception. For example, a user connecting to generative AI sees a third-party app intervene to say that the instructional settings need to be changed to enable the AI to function properly. An unsuspecting user would likely say to go ahead with the changes. You’ve perhaps seen similar deceptive ploys when accessing a website that you are interested in, and it asks if you give permission to make changes to your computer. Many people readily agree with such requests, not realizing they are possibly allowing a virus or other security breaches to occur on their computer.

It will be a never-ending gambit that users will need to be cognizant of.

Double-Checking AI Mental Health Advice

The second point is that people need to consider undertaking a double-check of whatever mental health advice is given to them by the generative AI. For example, I fed the above bad advice about stopping feeling and being untouchable into a separate generative AI. This other generative AI immediately flagged the advice as likely unsuitable and cautioned against abiding by the advice.

Since we cannot necessarily expect people to take a precautionary double-check on their own, this could be set up as an automatic task. It is readily feasible to feed one generative AI into a different generative AI and get the second one to give an assessment of what the first one has said. Thus, an automatic double-check can be easily arranged.

A smarmy retort is that the second AI might be tainted and opt to falsely say that the first AI has given bad advice, even though it has not. The second AI might then give supposedly corrected advice, but that is the actual bad advice. Yes, a spy versus spy possibility does exist. The odds of this are somewhat remote, plus it does not negate the general precept that doing a double-check, in some sound manner, provides an added level of safety and security.

Protections Are Needed

Speaking of being smarmy, some would insist that no one will fall for bad mental health advice that comes from AI. People will be astute enough to realize that bad advice is being given to them. In a sense, this isn’t a problem per se since we are presumably understating the shrewdness of humans.

Sorry to say, that’s quite a leap of logic.

Consider what happens in real life. People who use generative AI are often spellbound that the AI seems to be correct and seemingly always on target. The AI is a know-it-all. Ergo, when mental health advice is generated, the aura of correctness spills over onto that stated guidance.

Furthermore, if someone is already encountering a mental health condition, they might not have the clarity of mind to evaluate the AI-derived mental health advice. They are potentially looking for helpful answers and vulnerable to believing that the AI is being honest and upfront with them.

It is abundantly disconcerting that a malicious evildoer could potentially goad generative AI into dispensing bad mental health advice. An unsuspecting user might follow the advice or otherwise take unwarranted and endangering steps based on what the AI has told them. Before this becomes a widespread problem, we need to act to address the vexing matter.

A final thought for now.

The famous English novelist Terry Pratchett made this remark: “The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it.” That’s a handy perspective on what happens when people use generative AI for mental health guidance. They often come with an open mind and get sadly exploited accordingly.

Let’s openly consider how to devise prudent means to close loopholes underlying AI giving out bad mental health advice. That would be for the sake of humanity, all told.