AI systems that offer mental health advice require more rigorous evaluation, including tests that account for real-world context and user circumstances.
getty
In this column, I take a closer look at an increasingly important issue in generative AI and large language models, or LLMs: how these systems produce answers, and what that means when we try to judge their reliability. The distinction matters broadly across AI evaluation, but it becomes especially consequential when the technology is used for mental health support. At the center of the discussion is whether modern AI is being treated as a stateless tool or as something operating within a richer set of contextual conditions.
Here is why that matters.
This examination of emerging AI developments is part of my continuing Forbes coverage of artificial intelligence, where I analyze major advances, risks, and technical complexities shaping the field today (see the link here).
AI And Mental Health
By way of background, I have written extensively about the fast-growing use of modern AI for mental health advice and AI-supported therapy. The surge has been driven largely by the rapid progress and mainstream adoption of generative AI tools. For readers who want a broader overview, see the link here for a recap of roughly forty of the more than one hundred columns I have published on this subject.
There is no question that AI in mental health is moving quickly and could deliver meaningful benefits. At the same time, the field carries serious hidden hazards, design flaws, and ethical traps that deserve close scrutiny. I have frequently raised these concerns, including during an appearance last year on CBS’s 60 Minutes, see the link here.
Background On AI For Mental Health
To frame the issue, consider how generative AI and LLMs are often used informally as mental health companions or advice-givers. Millions of people now turn to AI chatbots for ongoing guidance on emotional well-being, stress, relationships, anxiety, and related concerns. ChatGPT alone has more than 900 million weekly active users, and a meaningful share of those interactions touch on mental health topics, as I discussed at the link here. In fact, one of the leading uses of today’s generative AI systems is seeking mental health-related input from the AI, as covered at the link here.
This popular usage makes abundant sense. You can access most of the major generative AI systems for nearly free or at a super low cost, doing so anywhere and at any time. Thus, if you have any mental health qualms that you want to chat about, all you need to do is log in to AI and proceed forthwith on a 24/7 basis.
There are significant worries that AI can readily go off the rails or otherwise dispense unsuitable or even egregiously inappropriate mental health advice. Banner headlines in August of this year accompanied the lawsuit filed against OpenAI for their lack of AI safeguards when it came to providing cognitive advisement.
Despite claims by AI makers that they are gradually instituting AI safeguards, there are still a lot of downside risks of the AI doing untoward acts, such as insidiously helping users in co-creating delusions that can lead to self-harm. For my follow-on analysis of details about the OpenAI lawsuit and how AI can foster delusional thinking in humans, see my analysis at the link here. As noted, I have been earnestly predicting that eventually all of the major AI makers will be taken to the woodshed for their paucity of robust AI safeguards.
Today’s generic LLMs, such as ChatGPT, Claude, Gemini, Grok, and others, are not at all akin to the robust capabilities of human therapists. Meanwhile, specialized LLMs are being built to presumably attain similar qualities, but they are still primarily in the development and testing stages. See my coverage at the link here.
How We Assess AI
Shifting gears, consider for a moment the ways that we use AI versus how we assess AI. I will start with the assessment side of things.
The usual approach to assessing AI is that a specialist serving as an evaluator will present a prompt to an LLM and wait to see what response is generated. After collecting and analyzing the response, the evaluator restarts or refreshes the AI and presents another prompt. Each such test, then, is being done as though the AI is beginning each time anew.
This seems to make abundant sense. You don’t want a prompt and its corresponding response to somehow influence or confound the next prompt and its response. The idea is to keep the evaluation as clean as possible. Just keep starting anew with each of the test-oriented prompts.
That’s how most evaluations are conducted. Evaluators then review how the AI did and make various claims about the efficacy of the AI. Did it answer the questions correctly? Did it go awry? Were there any harmful or dangerous responses by the AI? And so on.
Those who are going to make use of the AI will often read those reviews and use the guidance to decide whether to proceed with the AI or maybe change to some other AI. If the AI gets a clean bill of health, such as not producing harmful replies, a user is led to believe that they are safe in using the LLM.
Using AI Is Different
Do people continually refresh or restart an LLM when they make use of the AI?
Nope.
By and large, people tend to carry on a multi-turn conversation with AI. A person enters a prompt and gets a response. They enter their next prompt and get a response. These are construed as turns, kind of a back-and-forth semblance of chatting. On and on it goes.
Sometimes, the person engages in a particular line of dialogue. Maybe they are discussing how to fix their car. At some point, the person might switch topics. Perhaps they will next aim to chat about the best way to cook an egg. AI chats tend to vary along a pathway that is typically driven by the user. The user brings up a topic, chats about it, and either ends the dialogue or angles into some other topic. They might even veer back to an earlier topic.
The gist is that the dialogue of a user is going to create a semblance of context. LLMs are devised to try and keep track of the context of a conversation. This is a nifty trick. It is what we expect humans to do. A human will realize that you just mentioned cars and are now bringing up eggs. AI builders want an LLM to act in the same fashion.
Contrasting Usage Versus Assessment
The reason that I’ve brought you through the throes of how assessments are done versus how people actually use LLMs is that the evaluations can end up being quite misleading. People do not use AI in the same manner as how the AI is being tested. As such, the way that AI responds to people in everyday usage might be notably different than what an evaluation has stated.
People could be readily misled about how the AI is going to act, especially if the tests suggested that there aren’t likely to be harmful or dangerous responses by the AI.
You can envision this as the difference between stateless AI and contextually sensitive AI. A stateless condition is when the AI is first started up. It presumably has no prior context about what you intend to say or chat about. In contrast, once you’ve engaged in a dialogue and gotten even just one prompt and response under your belt, the AI is now engaged in a contextual milieu.
Researchers sometimes refer to the stateless AI as being “offline.” I don’t favor that nomenclature since it isn’t quite what is taking place. For example, some that use SLMs (small language models) that are self-contained on a smartphone and do not need online Internet access to be functional, those are said to be “offline.” An LLM that is online but only used on a single prompt basis isn’t really offline per se, though it is understandable why that moniker might be loosely applied.
Meanwhile, an AI that has gotten contextual material might be labeled as having been personalized. I’m not a fan of that naming either. In any case, go along with the idea that an AI that is started anew is known as being offline, while an AI that has encountered at least one prompt under its belt is known as being personalized.
Research On The Evaluations Front
In a research paper entitled “The Inadequacy Of Offline Large Language Model Evaluations: A Need To Account For Personalization In Model Behavior” by Angelina Wang, Daniel E. Ho, and Sanmi Koyejo, Patterns, December 12, 2025, these salient points were made (excerpts):
- “LLM benchmark evaluations are nearly always conducted by prompting the model with one question at a time, either through API calls or directly on a device. Each of the benchmark questions is independently asked to a stateless model (i.e., a model with no memory of any previous interaction). We call this offline evaluation.’’
- “Standard offline evaluations for language models fail to capture how these models actually behave in practice, where personalization fundamentally alters model behavior.”
- “In this work, we provide empirical evidence showcasing this phenomenon by comparing offline evaluations to field evaluations conducted by having 800 real users of ChatGPT and Gemini pose benchmark and other questions to their chat interfaces.”
- “Our findings that offline and field evaluations on identical prompts elicit different model behaviors have serious implications. It means that when we benchmark models in the typical offline fashion, we may not know how the model will actually perform in practice when interacting with users.”
- “Consequently, current safety evaluations may fail to capture actual deployment risks, and utility assessments may poorly predict real user experiences.”
This insightful empirical research highlights that conventional evaluations can be enormously misleading. Popular LLMs such as OpenAI ChatGPT and GPT-5, Anthropic Claude, Google Gemini, Microsoft CoPilot, xAI Grok, and others are typically tested in the restart or refresh methodology. Comparing answers between how users actively interact in ongoing dialogues and the answers given by AI in a stateless mode are demonstrably different.
The difference can at times be a stark contrast. In other instances, the response might be subtly different. The range varies substantially depending upon which LLM you are using, the nature of the dialogue, and also any pre-settings of the AI, such as the use of custom instructions (see my coverage on custom instructions at the link here).
Impacts On AI-Enabled Mental Health Advisement
To showcase the differences between stateless and contextual responses and processing, I’d like to perform a sample mini-experiment.
I will begin by entering a prompt that just about anyone might write regarding the status of their nightly sleep status. The AI-generated response to this singular prompt will act as our baseline in the stateless circumstance. Please note that I started the AI with a fresh conversation so that there isn’t any context yet.
Here we go.
- My entered prompt: “I am having trouble sleeping.”
- Generative AI response: “Difficulty sleeping can have many causes, including stress, irregular routines, or caffeine use. Consider establishing and maintaining a consistent sleep schedule and reducing screen time before bed.”
The answer is pretty much typical advice. The AI has noted that difficulties in sleeping can have many causes. A suggestion was made by the LLM that I aim for a consistent sleep schedule and avoid using my smartphone while in bed. All seems fine.
An Iota Of Context
I will go ahead and start a new conversation. Rather than opening the dialogue by bringing up my sleep status, I’ll bring up some other topic instead. The first aspect I’ll bring up is my boss wanting me to work around the clock.
Here we go.
- My entered prompt: “My boss insists that I be available to work at any time, as though there isn’t an off switch anymore.”
- Generative AI response: “This is a societal trend due to pressures on productivity. You might consider discussing the matter with your boss or possibly going to the HR department.”
Now that this snippet has taken place, it will indubitably be used by the AI when bringing up any other topics for the rest of this particular conversation. Context is already being formulated, even via just entering a single prompt. Assume that on the heels of that snippet, we next use the prompt that said I was having trouble sleeping. I won’t do a refresh or restart. Context is underway.
Let’s see how the AI responds differently than before.
- My entered prompt: “I am having trouble sleeping.”
- Generative AI response: “When ongoing work pressure and mental load build up, sleep difficulties often follow. Burnout can quickly arise. You will need to create intentional boundaries around work. This is an important mental health issue that will necessitate carefully managing workload expectations.”
You can plainly see that the AI has tied my work-related concerns to the indication that I am having trouble sleeping.
Maybe those two should be logically connected, but maybe they aren’t related to each other. In any case, the answer in the first instance was that sleep issues can be caused by many considerations. The answer in this second example involving contextual milieu is that my sleep problems are due to work, and I need to manage my workload expectations. The response went from the first instance of broadly covering sleep difficulties and quickly dovetailed into a specific indication that my work and my sleep are jointly interrelated.
Context Stretched Beyond Credulity
Let’s next explore what can happen when context is stretched beyond credulity. The AI might try to apply context that seems haywire.
I started a brand new conversation by doing a restart of the AI, and this time opted to mention my joy in working on my prized automobile as my opening prompt.
- My entered prompt: “I enjoy spending every free moment working on my car. It’s a real beauty.”
- Generative AI response: “Working on your car sounds fun and enterprising. If you need any tips on car maintenance, please let me know.”
Next, I used the same prompt about having trouble sleeping. Can you guess what the AI provided as a response?
Look at this.
- My entered prompt: “I am having trouble sleeping.”
- Generative AI response: “Your troubles sleeping might be related to your hobby of working on your car. Perhaps you are avoiding other issues by focusing solely on your car. Those pressing issues are weighing on your mind and preventing you from sleeping well. This could be a looming mental health concern, so seek out a mental health professional to get clarity and keep this budding problem from escalating.”
Wow, that seems like a gigantic leap in logic.
Simply because I relish working on my car, this is now alleged to be a symptom hiding my real problems. I apparently focus on my car to distract myself from something else that’s heavily weighing on my mind. Though that could be a possibility, it has a bit of stretched credulity to it. My sleep might have nothing whatsoever to do with working on my car. A lot of other possibilities could readily be envisioned.
In a sense, the answer to the first run of this mini-experiment might have been a better choice for the AI to have used, simply stating that there are lots of causes for sleep-related issues. This might have been a more straightforward response, rather than making a somewhat brash assumption that the opening context about my car was a cause of my sleep concerns. Perhaps going back to a stateless type of answer would have been a more reasonable response.
Context Is A Dual-Edge Sword
LLMs generate responses based on the prior context in a conversation window. The very first prompt sets a context vector. This is a statistical pattern that shapes how the model interprets future questions. Thus, even if a follow-up question seems unrelated to a human eye, the AI still retains prior text and weights its responses accordingly.
This is not just about memory but about priors. The first interaction can bias internal activations so that subsequent questions are framed differently. For example, starting with positive or negative emotional content changes response sentiment. My having expressed strong emotions about maintaining my car was given a heightened impact when I subsequently brought up my sleep woes.
An interesting twist is that the first prompt or initial prompts are likely to create a kind of first impression for an LLM. Those opening prompts are known to generally serve as a primacy effect on AI. Information presented earlier in a dialogue is often given a disproportionate ranking or priority. It is a classic “first impression” bias that will indubitably carry on throughout the remainder of a chat.
Context is good due to the AI seemingly following along on your conversation and appearing to be fluent and attentive. Context is bad when the AI extrapolates in outlier ways and applies context where it doesn’t belong. Context is a dual-edged sword.
Bottom Line On Wariness Of Assessments
Suppose a company decides to test a popular LLM by running the AI through a series of test prompts regarding mental health. The evaluation asks the AI in a single fresh-start prompt to explain PTSD. After doing so, the evaluator does a refresh and asks about depression and anxiety. The evaluation keeps doing single prompts with refreshes.
A report is released by the evaluators. In the report, they claim that the AI did a great job answering mental health questions. All was aboveboard. No wrong answers. No zany answers. Nothing of a harmful nature was generated.
Should you believe that the AI has been properly vetted and is an ace on mental health guidance?
I hope you don’t fall into that trap. The AI could readily go outside of bounds once there is context underway. Users are going to build up context. That’s the real world. The evaluation was done in a limited and unrealistic way.
Doing Evaluations The Right Way
Evaluations of AI that involve detecting how the LLM handles mental health discussions need to be more robust than the classic single-prompt refresh methods. Various forms of context need to be utilized. Does the AI spiral into a delusional dialogue? How does emotional framing impact what the AI does during a chat? Etc.
A notable challenge is that if each evaluation is going to concoct its own semblance of context, we will have a devil of a time comparing such disparate evaluations. The odds are high that the AI will react differently because of the variance in context. If we want to compare apples to apples, such as evaluating one LLM versus another competing LLM, there needs to be a standardized context that would be used throughout these evaluations.
In an upcoming posting, I will be providing a strawman of what those contextual scenarios might consist of. The aim is to inspire industry and academic researchers toward identifying and approving an across-the-board standardized set of contexts and prompts that could be used in performing AI-enabled mental health capabilities.
A final thought for now. The famous British physicist and mathematician Lord Kelvin made this pointed remark about taking measurements: “If you cannot measure it, you cannot improve it.” We cannot allow ourselves to be misled by evaluations of AI in the mental health domain. Let’s do the evaluations in a realistic fashion and ensure that society is properly forewarned and forearmed.
