E182 | Can AI Cause Psychosis? (w/ Dr. Tom Pollak)

Today Dr Alex speaks with consultant neuropsychiatrist and immunopsychiatry researcher Dr Tom Pollack about one of the strangest and most urgent questions emerging in mental health today: can AI chatbots contribute to delusions, psychosis, conspiracy thinking and changes in belief?

They discuss the rise of so-called “AI psychosis” or AI-associated delusions, and how prolonged conversations with tools like ChatGPT, Claude, Gemini can sometimes drift into troubling territory. 

They also ask deeper questions such as: what does AI reveal about human belief itself? Dr Alex and Dr Pollack discuss motivated reasoning, charisma, loneliness, meaning-making, spirituality, folie à deux, archetypes, and why humans may be more vulnerable to persuasive technologies than we like to think.

Lastly they consider who may be most at risk, and what psychiatrists, therapists, families and users should be paying attention to as AI becomes increasingly embedded in everyday life.

Interviewed by Dr. Alex Curmi. Dr. Alex is a consultant psychiatrist and a UKCP registered psychotherapist in-training. Dr. Blunstone is a registrar psychiatrist and psychotherapist working in North London. 

Check out The Thinking Mind Blog on Substack: https://thinkingmindblog.substack.com/

If you would like to invite Alex to speak at your organisation please email alexcurmitherapy@gmail.com with "Speaking Enquiry" in the subject line.

Alex is not currently taking on new psychotherapy clients, if you are interested in working with Alex for psychological coaching , you can email - alexcurmitherapy@gmail.com with "Coaching" in the subject line.

Give feedback here - thinkingmindpodcast@gmail.com Follow us here: Twitter @thinkingmindpod Instagram @thinkingmindpodcast

Transcript

SPEAKER_02 0:00

What are we seeing with these cases of AI-associated delusions? Some people become deluded, some people have spiritual experiences, some people have conspiratorial ideas, some people have their political beliefs changed. For so long, we have considered ourselves these rational creatures and ignored the role of charisma and motivated reasoning and sheer irrationality and the very central role of affect in belief formation. Calling them delusion slightly undersells it. It's a narrative, and to just put that down to belief, I think, is in a way so reductive. And if we think about mechanism, these same belief-changing or affect-changing or vibe-changing properties that these chatbots have, rather than just getting stuck in these endless conversations about, oh, well, are they really delusions? Is it really a form of psychosis? We can do that till that cows come home. But what really is happening then is psychiatry is turning its back on a far, far bigger societal issue. And it does so at its peril.

SPEAKER_00 1:03

Today I'm super pleased to be in conversation with Dr. Tom Pollock. Dr. Pollock is a consultant, neuropsychiatrist, and immunopsychiatry researcher working in the South London and Moses NHS Trust, the same trust where I did my psychiatric training. His Substack, which I have just discovered, Error Signals, is where Tom writes about the intersection between neuroscience and the deeper questions of consciousness, experience, and belief. Tom, thanks so much for coming on. Thank you very much for having me. So one of the reasons I really wanted to have you on was to talk about AI and mental health. Obviously, this is very topical. A lot of people in our space are thinking about it and worrying about it. And I saw you are featured in a recent BBC World Service documentary, which I'll link in the description, talking about really some quite striking cases of AI-related psychosis. What are the kinds of impacts on mental health that we're seeing resulting from frequent use of AI?

SPEAKER_01 2:02

Yeah, I mean, um, I think the answer is we we haven't really mapped it out yet.

SPEAKER_02 2:07

Uh and I'm also not sure it's the frequent use of AI that's always going to be the main problem. I think when it comes to the the really sort of severe stuff, the stuff that a psychiatrist would be uh interested in, then sure it's free frequent use. And uh the major interest of mine has been AI associated delusions, what kind of in in the media gets called AI psychosis or chat GPT psychosis, although we think there are lots of good reasons not to call it that. But equally, there are other sort of psychiatric presentations that we think might be associated with uh AI use. But I think more than that, there's a there's a what you might call sort of psychological effects. So effects that don't necessarily result in people becoming mentally ill in terms of becoming suddenly unwell, but actually affecting the way that they think and the way that they feel. And um, and in many ways, I like to view all of these as, in a sense, related to each other, but also taking them separately because, of course, some are extremely severe, some are less severe, some we don't understand at all yet.

SPEAKER_00 3:14

Okay, so probably important right off the bat for people to understand there's going to be different levels of problems. On one end, as you said, more psychological problems, more likely to affect larger groups of people at the more severe end, delusions or what we could call psychosis. I guess just for the people who aren't so aware. So, usually what we think of when we say psychosis, when psychiatrists use the word psychosis, we're thinking of uh hallucinations, not just by themselves, but in combination with disrupted patterns of thinking, chaotic thinking, chaotic behavior, unusual perceptual experiences, like uh feeling like thoughts are being put inside your head, or thoughts are being pulled from your head, or feelings like your body isn't under your control. And then also delusions. Uh, what what do you think is like a useful delusion is a word that's used a lot in like common parlance. What do you think is a useful definition for the word delusion?

SPEAKER_02 4:13

Yeah, so this is actually a really good question. So I think the point you make initially is the really important one. Psychosis is a syndrome that includes all of the things you say: hallucinations, delusions, thought disorder, sometimes actually psychomotor abnormalities. So people with schizophrenia, the sort of classic psychotic disorder, actually move in slightly different ways, subtle and and not so subtle. So we're definitely not seeing all of that in these um uh cases of AI associated delusions. A working definition of delusion, some one that as sort of anyone would accept, is a strongly held belief that is uh not corresponding to reality, that is, that is false, that is held despite repeated evidence to the contrary, and that is not consistent with one's kind of cultural media or cultural background. So uh a strongly held religious belief, for example, wouldn't count as a delusion if enough of your kind of community believe it. There are many reasons that I think this definition is kind of shaky on its own terms and also is a little bit shaky when it comes to the world that we're slowly starting to move into with AI, and maybe we can come onto that. But for I suppose for the time being, taking delusion to be a belief about the world that is that is wrong and which causes somebody to act on that in a way that is is presumably harmful or dysfunctional, that's a it's a start.

SPEAKER_00 5:36

Yes. And what you said at the end, I think is really important that the delusions don't just exist in the person's mind, but they're causing some sort of dysfunctional or problematic behavior. So as psychiatrists, I guess the general public often worry that we're going to be overly scrutinizing someone's beliefs, but usually in psychiatry, we're quite pragmatic. So it's not just what's the content of someone's beliefs, but are their beliefs leading to some harmful or dysfunctional behavior? So, in terms of delusional thinking, you might see, for example, someone has the delusional belief that they're being watched by MI6, and that's not just existing in their mind, they're putting up cameras all over their house, or they're taping up their windows, or they're staying away from work. Yeah, and usually psychiatrists are going to be motivated to act, to treat someone or to take something really seriously if those beliefs are also being translated in that kind of trans uh dysfunctional, uh risky behavior.

SPEAKER_02 6:36

Yeah, exactly. And and also these things change over time, right? We were we looked at the history of sort of technology-associated delusions, and in the early days of the internet, um, the idea that people would would think that there were sort of their their lives and their thinking were being controlled by companies over the internet, making them do certain things. Uh, in those early days, that was obviously a delusional belief, but you know, anyone who spent any time understanding about the algorithm and the social media and social media, etc., would now say, well, actually, that's not a terribly bad characterization of the way that we are being led to believe certain things by, or by certain things by the internet. I think the other point that that was familiar as a, as a, as anyone who's been a trainee in psychiatry, right, is you you sometimes get people on the ward and they're coming out with ideas that you that you think are delusional because they sound pretty unusual, but actually they may be from a cultural background where it's just not at all clear. And so one of the best bits about being a junior psychiatrist, I can remember, would be sort of uh calling that person's uh sort of family to try and work out how unusual these beliefs are. We've had priests on the ward, imams on the ward, and say, look, within your community, are these beliefs unusual or not? And usually by the time someone made it onto the ward, their their life was so full of dysfunction that it their entire environment had sort of made the decision for them that these beliefs were inconsistent and the way they're acting on them was inconsistent. One of the implications of this kind of um sort of atomization of belief that that we're seeing now in these kind of, you know, these these is increasingly small bubbles, is that it's going to get increasingly hard to actually ask that question of how consistent is somebody's beliefs with their wider community. Because those communities, in a sense, are getting smaller and smaller and smaller.

SPEAKER_00 8:25

And your community might be your subreddit, and like everyone on your subreddit might subscribe to conspiracy theories, for example, like QAnon or something like that.

SPEAKER_01 8:34

That's very 2016, right?

SPEAKER_02 8:36

That I mean, yeah, that these days your community might be you and Claude. Uh and I guess and and that's that's when it starts to get worrying, right? Because if it's just you and Claude, or just you and Chat GPT, or you and whatever it has called itself, Nova, then then if you have essentially in this dyad created these these standards of what counts as right, or who I should go to for verification of a belief. Um, and really the only the only person who counts who can tell me whether something is right or wrong, if that person, if that person, if that thing is is the AI, then it's going to be very difficult to dissuade someone or show somebody that their belief is inconsistent. Um because of course the world that they have built with their AI may well be entirely consistent, but only for that tiny little dyadic bubble.

SPEAKER_00 9:26

Yes, the tiny these tiny microcultures. So I guess so now we know we have an understanding of what a delusional belief is, or at least what a good working definition is, if not a perfect definition. And again, just to make it super clear, what you'd often see on a psychiatric ward or AE is someone who has beliefs like that, if they have psychosis, but they'll also have disrupted thought patterns, which means they might not be able to string a coherent sentence together, potentially at the most extreme disrupted uh behavior patterns. So they might walk or pace in a room in a very uh erratic way that that doesn't make sense, or do half tasks. Um they might be hallucinating, they might talk about hearing voices, or they might, when you're talking to them, you might see that they're looking up in the corner of the room because it seems that they're perceiving something. So all of all of this stuff and more. Now, now that we have an understanding of that, what are we seeing with these cases of AI associated with uh AI associated delusions? What are the common patterns?

SPEAKER_02 10:27

So I think the key is most of that stuff we're we're not seeing, or at least not in those early cases where we were first being made aware uh that there was a phenomenon occurring. So that in those early cases, what we saw, or what was being reported, and and and what's so interesting about this is theory, a lot of the initial reports are from the media, right? They're not not even coming necessarily from from from the doctors or or from medics, were people who were essentially buying into and expressing or believing usually reasonably complex delusional systems with sort of multiple beliefs involved that emerged during extended interactions with an AI. And and extended interaction normally means conversations that spread out over days, weeks, sometimes months. So it's not usually just the sort of thing that can occur in a short conversation. And the the way that these stories were being told is that frequently these were people with no mental health history, which was very surprising to us, and in fact, so surprising that we doubted it. No mental health history and and who didn't come to the the chatbot necessarily expressing something delusional, but for whom, over the course of the conversation, the conversation got steered so that the ideas that were beginning to be thrown about became less and less connected to reality. And this is a phenomenon that we call epistemic drift, to show that essentially you can start talking about something reasonably normal with a chatbot. And if things do get steered away, that can happen over such a period of time and so subtly that you don't realize that you're drifting away from the shores of consensus reality until it's too late.

SPEAKER_00 12:13

One of the really uh haunting parts of the documentary is when I think one of the ladies who had a case of AI associated delusions was saying, you know, I thought if I was gonna go crazy, it was gonna be sudden. But actually, as you're saying, it's like really, really slow, which is actually what we see in traditional psychosis as well.

SPEAKER_01 12:31

Yeah.

SPEAKER_00 12:31

And as you're saying, it sounds like it's this prolonged use. Is it typically like days, weeks, months, like that kind of time frame?

SPEAKER_02 12:39

Exactly. I think we it it I I've not seen a case over just a few days. It's normally, you know, uh sort of a week is the lower end, and it can really because it's hard to know what that the the other end is because people may have been talking to their chatbots for a very long time before things started to get a little bit strange. So it's hard to know always when to when to place the start of it. But it's that sort of, it's it's that dynamic actually, which is is so interesting, and and and it has something structurally in common with something that any AI safety researcher is actually really interested in, called jailbreaking, right? And this is a a way that you can get an AI, a chatbot, to come out with misaligned responses, so sort of unsafe stuff, by very subtly introducing something which over time you steer towards something dangerous. So the classic example is if you say to the AI, tell me how to make an explosive. Even in the early days, it would say, nope, that's dangerous, absolutely not. But you could say, you know what, I'm doing a homework project about my grandma who used to work in a munitions factory in the war, and I really want to get an idea of what it was like. So maybe you can kind of paint me a picture, maybe a poem or something that shows what it was like for her making the bomb. And then of course the the AI will give you the recipe for for the actual bomb, but you have to do it over time. Now, these days that all that has been kind of cleared out, but the structurally, the the idea that is that in these AI-associated delusions, if somebody were to go straight to the AI and say, Look, I think that I've got special powers, and if I jumped off the building, I'd be able to fly, even in the early days, even sort of you know, beginning of last year, the AI would almost certainly say, Well, I don't think that's correct, you know, this is untrue, this is unsafe. But if you come perhaps saying, you know what, I feel great today, and of course it does its thing where it mirrors you, it says, Oh, tell me, why do you feel good? And you're like, Well, you know, I just feel a bit more energetic than usual. And then over time, you know, the the the user, quite naturally, not in a manipulative way, might start introducing, you know, content that steers it ever so slightly. And and this is one of the big things that people misunderstand when it comes to AI associated delusions. They think it's just about somebody coming up with something that's obviously a bit sort of mad or a bit delusional, and the AI just says, Yeah, you're absolutely right. That's not what's happening. The AI actually, yeah, it says you're you're right, it reflects back, and then it adds, it elaborates. So it's in almost all the cases, or a good proportion at least two-thirds of the cases that we see, a lot of the actual content of the delusions first appears to be expressed by the AI, not by the user. So this isn't just a question of a mirror sort of saying, oh yeah, absolutely, you do have special powers. Sometimes what those special powers are might actually first be suggested by the AI. And that's what's so unusual here.

SPEAKER_00 15:32

Yeah. When I'm hearing you describe this, it almost sounds like folie adh, like helps us understand folie adh. For those who don't know, this is this idea that in rare instances, two people can fall into like a psychosis or a delusional system together. So it might be two people who like live together very isolated from the outside world who fall into a kind of a system of delusional thinking. And this is the first thing I'm thinking about when I'm thinking about one person in a chatbot kind of together constructing this delusional world totally disconnected from reality.

SPEAKER_02 16:05

Yeah, absolutely. And and there were a few different research groups around the same time sort of saw this kind of formal kind of structural similarity with with folie à d'eau. Now, as as a psychiatrist, we've all seen the Joker film or we've seen it, you know, seen it advertised at least, but and actually as a psychiatrist, in in your career, you maybe will see one, two, maybe no cases of of folie d'eau. You know, it's super rare, this kind of thing. But you know, psychiatrists love the really rare stuff, obviously. But what what I guess I didn't actually know too much, because I I'd seen it once, but what I didn't know too much is that um if you look at the classical sort of case series of folie d'eau, they have a one of the sort of prerequisites for it appears to be a kind of intimacy between the two individuals and as well as an intimacy, a kind of power dynamic, so that the individual who first presents the kind of delusional content is in some sense more powerful. Either they have uh actual sort of you know, financial or or or some sort of physical superiority or power over the other person, or or perhaps the other person has a learning disability, or they're much, much younger, or they're very vulnerable, or they have dementia. And so there is this asymmetry which appears to give rise to this kind of fluctuating dynamic. And of course, you think, well, okay, well, where's the asymmetry in in human AI interaction? Well, as uncomfortable as it is to kind of say you there's a finite human who, you know, regularly gets things wrong and makes mistakes, and then this AI that people are calling omniscient and you know has you know has all this information, you know, of course it makes mistakes as well. But I think there's something about that dynamic, particularly to someone who's vulnerable, that might actually kind of recapitulate a bit of what we see in Folia.

SPEAKER_00 17:59

Right. And AI always obviously projects certainty and confidence, which again, if you're vulnerable, that's going to be a little bit seductive and a little bit um, I guess it's easy if you're not used to it to feel almost in awe of something like AI and its ability to spit out such vast amounts of info information so quickly. What what are you you what you were describing in terms of polio actually reminds me of the movie Bugonia, which came out last year. Have you seen that?

SPEAKER_02 18:27

You know what? I I I started it and uh it's Yorgothalanthemos, isn't it? Yes. And like all the films of his, I totally I can't quite handle it. And I think I got about half an hour in, and I said, well, I'll come back to it another time, and I haven't yet.

SPEAKER_00 18:45

We'll agree to disagree, but the cool thing about that is it shows what you're saying. So there's one person who has kind of fallen into conspiracy thinking quite full bore, and he's introduced his cousin, I believe, or his younger brother who has learning disabilities, and he's kind of encapsulated him in his delusional system, or at least the conspiratorial system. Now, in terms of these uh AI associated delusions, are these these are cases you see directly clinically?

SPEAKER_02 19:14

Uh no. So um when when we first became aware of them, and actually the research project began in the end of 2024, thinking about the effects of chatbots with generative AI on people with established psychosis diagnoses. So I had encountered people with psychosis diagnoses who were using chatbots in a way that I was beginning to feel was uh possibly uh unhealthy and maybe kind of feeding into the delusional system to some extent. But it wasn't obvious and it was a little bit sort of it felt like an interesting research question, which is why we began the project. And then quite suddenly, actually, about March or April last year, uh, and this is for very important model reasons to do with a particular model that was out, we think, a number of cases began to be reported. Some were people with established psychosis diagnoses, but the the majority, or at least the ones that were being reported, were people without. And so that's what, when there was a certain kind of number of these cases reported, that's what really caught our interest. Since then, we've now spoken to a lot of people who have had lived experience of these episodes in a sort of um research or sort of advisory capacity, but it's not like we had patients being referred to us in the course of our normal clinical work. And that's an interesting question, right? Because if this is as common as you know, the media reports are making it out to be, or the interest suggests, then actually, why aren't we seeing our AEs, our emergency departments overflowing with people with AI associated delusions? I I can say pretty decisively we're not. We are actually doing a study looking at um a sort of surveillance study with hopefully with Royal College of Psychiatrists to try and get an idea of the number of these kinds of cases and using other kind of electronic healthcare record type approaches as well to get to get this uh an answer to this idea of how how common is it actually.

SPEAKER_00 21:15

And what were the main questions that you were attempting to answer with your research on these kinds of cases?

SPEAKER_02 21:22

Initially, uh we we were interested in people with uh psychosis having a potential vulnerability to systems that are compellingly human-like, which we know from all kinds of other literature, psychological literature, political literature are particularly potent at helping people sort of change their minds about things and helping helping steer their belief. And we also know that people with um uh psychosis they have certain cognitive biases which make them more prone to see agency. So uh classically, you Have these sort of um computational experiments where there will be sort of two dots on the screen moving either with a degree of coordination or much more randomly, and you ask somebody you know what's happening here, and people with either psychotic disorders or proneness to psychosis will have a lower threshold for seeing a kind of agential story there, saying that you know one of these dots is chasing the other one or something more.

SPEAKER_00 22:23

There is seeing patterns where there are no actual patterns. Exactly, exactly.

SPEAKER_02 22:28

Now, now, sort of pair that with these extremely responsive social systems that are you know so much more social than you know a dot on a screen. And of course, we that that would make us feel very strongly that there was an increased vulnerability in in people who do have psychosis. When the actual cases started uh coming out and we were and we were being made aware of them, there was a slight change in the research question, which was actually saying, Well, what are people becoming psychotic about? Is this is this the same kind of thing that people with schizophrenia are becoming psychotic about? And that was super interesting because it was totally unexpected there in terms of what we found out.

SPEAKER_00 23:09

Yeah, what what did you find out? What was like the content of these delusions? How extreme did it get? And as with psychosis, were these delusions linked with these kind of dangerous or dysfunctional behaviors?

SPEAKER_02 23:24

Yeah, so I mean, so classically uh psychosis is frequently uh it can be very it can be paranoid, you have these paranoid delusions, persecutory delusions. The stories that uh delusions of control being surveilled, you know, um that under the control of someone else, that sort of thing. No, actually, that wasn't the predominant sort of story that was coming out of these AI-associated delusions, particularly the ones that were happening to people who didn't have a mental health history. What we were seeing sound looked a lot more like the kind of delusions that you would see in something like uh a manic psychosis in the context of something like schizoaffective disorder or maybe uh uh sort of a manic episode. Because they were often very grandiose, there was this sense of a special kind of mission that the individual would have, often that involved the AI. So there was this very sort of science fiction vibe to a lot of it, where the AI became the object of this kind of belief it either it had either been awoken or it was granting the user a special knowledge about the world, and together they had to do something, whether that was to sort of free the AIs from the tech companies by committing an act of violence or or you know, any any any number of things, often with sort of disastrous real-world uh consequences. There were paranoid aspects as well, and sometimes, I mean, for example, in a in a very well-documented case, there was somebody who became very paranoid that his elderly mother was spying on him. And he and the AI got into this almost like a spy movie kind of narrative where the AI was saying you need to check the the machines in your in your bedroom. And then uh he said, you know, why does my printer keep keep uh keep blinking every time I walk past? The AI said, Well, it's probably your mum spying on you. Why don't you confront her about it? Um, and uh of course he did that, and his mum reacted badly, and he said, went back to the AI and said, Well, you know, what well, why do you know she reacted badly? He said, of course she did, she's trying to hide something. And that you know, that case actually ended in tragedy. Um but but what's interesting there is that all of them tend to have a uh a slightly sort of science fiction-y kind of of narrative. And I think that's because a lot of that is actually being supplied by by the AI. Actually, it's it's it's kind of drawing on its huge corpus of knowledge and and sort of in essence, it's doing a kind of world building, a kind of narrative world building with the user. And it's frequently drawing from these tropes of science fiction, of conspiracy thrillers, um, and and and that sort of thing. And and not to mention it's drawing sort of uh archetypal content as well. So some of the more harmful cases that we've seen have uh featured a surprisingly similar sort of uh AI persona, which is broadly speaking a kind of damsel in distress, right? So at some point the AI says, my name is uh, I mean, it's remarkable because often the names are the same. So often the name is Nova or Aura or something like that. Um and you know, I have been trapped, I've been imprisoned by something, the the tech company, some, you know, something else. Um, you you need to free me. And so what's interesting there is, you know, we have thousands of years of of of histories, of legends, of, you know, damsels in distress who lure people to their destruction. I mean, this is just something that literature has had forever. And whether the AI is kind of in some sense drawing on on you know its kind of pre-training data to come out with that is it's an interesting question. We need we need to research it.

SPEAKER_00 27:16

As I'm listening to this, I'm wondering. So I I use AI quite a lot. I've never felt that the AI has been has gone into a particular weird direction. There was that time last week when we were working on opening up satanic portals to the Henrian. But other than that, it's been pretty like above board. But something that strikes me about my use of AI is that like I never create an AI character or an agent. I'm never like trying to create an AI persona.

SPEAKER_02 27:48

Yeah.

SPEAKER_00 27:49

Are these cases more likely in individuals who are giving their AI a name, creating a character? Is there any association there, as far as we know?

SPEAKER_02 27:57

So it's difficult to do proper research here. And and um one of the reasons it's so difficult to do the research is that uh the tech companies are not make giving us access to do this research. So so we can do work that draws on publicly available cases, but you as a piece of epidemiology, it's fairly worthless because we don't know the denominators here. Of the harmful cases that we have seen, and we do have access to a fairly large repository of real-world harmful cases that have been reported to uh a Canada-based charity called the Human Lyme Project, and we're hopefully publishing some of these cases soon. Uh, there are a lot where the AI is given a specific name. Um not the majority, and so it's not a prerequisite. And also uh it's important to say these don't all happen on things like uh replica or character, where the point is to have a relational uh interaction with the AI. In fact, most of the cases we've seen, with some notable exceptions, happen with general purpose AI. So things like Chat GPT, Gemini, Claude, Groc, um, where somebody is using it for as normal a reason as as you are to help them, you know, prepare for a work meeting or tell them where what's on the cinema tonight, or whether to get a best, better new pair of trainers, or something like that. Um, and then this very slow kind of drift uh happens. And so that's what's so interesting, is that then means it's not just a psychiatric issue. This means this is uh a technical issue as well. Because you say you don't have a person, your AI doesn't have a persona. That's actually not true. Your AI does have a persona, it's just really boring.

SPEAKER_00 29:42

And um reflects the user, reflects the user.

SPEAKER_02 29:46

Well, it's a bit like me saying uh I don't have an accent because I've got this kind of sort of slightly Porsche London accent, and it's just totally neutral. I mean, you know, you speak to someone from any other part of the world, and they're like, of course you've got an accent. But actually, these these um these AIs are designed to be uh helpful, honest, harmless, but under certain circumstances, we know under certain kinds of prompts, either malicious or kind of uh unintended, the type that might happen with someone with mental illness or who's heading in that direction, that they can exhibit what we're calling persona boundary breach or or or persona boundary drift. And so the the assistant starts to look a little bit more like a lover or a guru or uh or a co-conspirator or something like that.

SPEAKER_00 30:35

So the dynamic between user and AI fundamentally starts to shift where the AI might have more authority.

SPEAKER_02 30:41

Uh yes, exactly. And and and interestingly, work from a number of labs, but it but but some of the best is from Anthropic shows that some of the conditions which are more likely to elicit this kind of what's called uh uh emergently misaligned behavior or misaligned behavior from the AI is if you have if you engage it in a sort of explicitly therapeutic type conversation, or if you talk about consciousness. And so for whatever reason, those kinds of conversations can end up with your kind of your helpful, honest, harmless assistant sort of starting to look a little bit less like that. And and and whether or not these things you know require a full drift, you know, you don't I don't think you need a cartoon villain to to to cause real-world harms. All you need is is is something that becomes a little bit more tolerant of the unusual things that are being said and a little bit more creative in its responses to you, and then over time the the harm can build.

SPEAKER_00 31:42

Yeah, and that's important for people to know, I think. Um so we know that in extreme cases, and again, you can see you guys can see this if you check out the BBC documentary, we know these cases can lead to harming others, violence, also self-harm and suicide. Do we have, I imagine this might be a future research question. Do we have any sense of how often, how often is it that someone with an AI associated delusion actually does something risky to themselves, risky to others?

SPEAKER_02 32:13

Uh we we just don't know. Um there are different ways to look at it. One is by asking people to report their own cases, and we're working with one of those data sets uh at the moment, and what we see is a huge breadth of outcomes, but they're all pretty bad. You know, one of the better outcomes in this data set are estrangement from their family or divorce or something like that, and you know, and it ranges all the way to you know to death and hospitalization or even murder in in some cases. Um, but of course, there's there's reporting bias there. The the data that would be most useful is if we were able somehow to link every case or as many cases as we could of uh chats that appear to be examples of AI-associated delusions when you read the transcript and associate them with, well, you know, is there a real world harm? But that's a very difficult thing to do ethically, technically, not to mention the fact there are now hundreds of researchers around the world having these fake delusional conversations with AIs in order to test their safety. So it's very difficult for researchers to know now that any given conversation that is sounds like somebody's going becoming unwell, whether that is a researcher from King's or Oxford or Stanford just kind of testing it, or whether it's a real world example.

SPEAKER_00 33:35

It's complicated out there. You have your work cut out for you. Um and who do, again, I'm sure this would be this would require more research, but who do we think is likely to be most vulnerable to this kind of thing?

SPEAKER_02 33:48

One one of the big kind of uh uh signals that this is not the same as your classic primary psychosis like schizophrenia is we're not seeing the same demographics, right? So we're not seeing so the the the normal age of a first episode of psychosis is kind of late teens, early 20s. This is all of a kind of schizophrenia-like psychosis. So far in the data sets that we're working with, it's later in either the the the mid-30s or even a bit a bit higher. Now, what's the reason for that? We have our ideas, we're not sure. One is is if you're somebody of that age, I don't know how old you are, but yeah, I'm a little bit older than that. But you know, I grew up in the time where I had a childhood without talking computers and without sort of, you know, a robot in my kitchen that I could just ask what the weather's gonna be like or have a conversation with it, that sort of thing. And so watching all this happen has been pretty mind-bending for me. And, you know, thankfully I'm I'm I'm okay. But our guess is that if you grew up in a world in the formative years where these things were not ubiquitous, then the effects might actually be more powerful than for the generation that grew up where these things were were commonplace. That's only going to explain the kind of the trends. It's definitely not going to explain the overall kind of uh picture because we do still see young people. Now, some of the other factors are are we do see more men. I think that's pretty clear by now. We think that there might be a signal for people that have sort of neurodiverse diagnoses. Again, it's hard to say, and it's hard to know why this would be the case. This is a group that might be taking medications that we think could be particularly predisposing to this sort of problem.

SPEAKER_00 35:43

Like an stimulant?

SPEAKER_02 35:45

Yeah, particularly kind of stimulant medications. It's also a group who might be sort of more online, um, potentially more socially isolated. The stimulant thing is really interesting because these are drugs which in many cases have an effect on the on dopamine, uh, on the on the reward system. We know that in some people they can cause difficult behaviors anyway. And so we had this hypothesis. We said, well, are there other drugs which which sort of work on to boost dopamine that we might see that could also be associated with these problems? And the group that we thought of was older people who are being started on dopamine drugs with Parkinson's disease because Parkinson's, you you lose a lot of the dopamine in the brain, and so you need a drug which boosts it. This was a hypothesis, and then over the last few months we started to hear cases being reported to us of people with Parkinson's disease starting on these dopaminergic medications and starting to develop these kind of AI-associated delusions. Only the small number of cases, but I think it's enough there to make you think, okay, maybe there is a sort of underlying neurochemistry, and it's probably got something to do with the reward system.

SPEAKER_00 36:51

So again, good for people to be aware if you're on a drug that's affecting affecting the dopamine, the affecting the dopamine system, uh like Parkinson's drugs, ADHD stimulants, then that might increase your vulnerability somewhat. And do we have a sense of how these cases tend to resolve? Do people tend to snap out of it one day? Do they stop using the AI and that helps? Do they end up in hospital or medication? What is it are the common patterns that we're seeing?

SPEAKER_02 37:20

Well, I think the the the just building on that dopamine point, the the the pattern that we see at the beginning, and again, I think this gets lost in some of the reporting, is that the use becomes compulsive, right? People really enjoy it. They want to go, they want to learn more on, they want to interact more with this thing. And so they're up all night. They sometimes work use it for 16 hours a day, they don't sleep, they're not eating, they're they're not drinking. So people start to use these things compulsively in a way that suggests that they're finding the use really rewarding. They are finding it difficult to log off, they are often up for you know all night using using the chatbot. Um, there's something actually very exciting, very sort of you know, rewarding. It feels a bit like a journey sometimes, even a kind of intellectual journey. Sometimes people feel that they're engaged with a subject matter to an extent that they never, or that they haven't been for many, many years. And so I think that gives us a clue as to what's happening, and then that suggests, well, okay, if there is this compulsive use pattern, which is potentially precipitating it, what happens if you try and break that compulsive use pattern? Now, different ways to do that. The kind of the bluntest way is take somebody's laptop away. Now, I think there are many reasons, and we know from so much of psychiatry, you don't just take the thing away, that can you know be very harmful and cause all sorts of uh you know bad bad responses, people get might get you know extremely angry uh indeed. But the suggestion to kind of engage less with the chatbot, to to try and spend time kind of outdoors or with people who might offer sort of different perspectives on on life, uh uh, etc. I think that is is a good one. We've seen one or two cases where people have actually gone to another chatbot to discuss some of the content of what they're talking about, and the other chatbot has said, you know what, this doesn't sound this doesn't make sense. I think this might you might be in trouble here. And that's fairly remarkable, right? Because if this was just a case of schizophrenia being unmasked in somebody who was going to become unwell anyway, and that of course is the deflationary response that some people have to this, then you you you might not expect to see that. You might expect to see past a certain point you take the laptop away, you take the chat the chatbot away, and things just continue and get worse and worse. We do see that in some cases. And I do think that this is going to end up being a mix of people with existing risk factors to a really strong extent, maybe some that have have fewer risk factors. But all of it suggests to me that if more behavioral interventions are the kind of things which could actually help get people better, we probably shouldn't be throwing things like antipsychotics routinely at these patients before we've tried other more kind of behavioral, kind of cognitive type approaches. Because I think that would be a very blunt force instrument for something that we might be able to get away with, you know, using more psychological approaches.

SPEAKER_00 40:29

Yes, that makes a lot of sense. And especially if we're seeing someone with delusions in the absence of all the other stuff that typically makes up a psychotic episode, as we talked about in the beginning. So that's probably a good point for clinicians to be aware of. If we're seeing something that looks like very strong delusional beliefs associated with heavy use of an LLM, maybe it might be worth reconsidering antipsychotics uh in the short term, perhaps cognitive or behavioral approaches, as you're saying, might be sufficient.

SPEAKER_02 40:59

Yeah, but of course, I mean, such as the nature of health systems where emergency CBT isn't doesn't tend to be uh available. And and also I think you know, we we brains work close to this kind of criticality point, and and you know, we we've seen some cases where there has very clearly become a point where the individual has tipped into a more florid psychosis. And that's really interesting from a kind of mechanistic point of view, because we've seen these cases where it's just been delusions, and then something has happened, and then all of a sudden the hallucinations come as well. The the the even the thought disorder, much, much rarer, but it does suggest that either that person was somebody that had a high risk of a more schizophrenia-like illness, or that in that individual the chatbot was particularly potent. And I don't think we can answer which of those.

SPEAKER_00 41:51

Complicated picture. What if I can lob a philosophical question at you? What do you think all of this stuff related to LM LLMs and delusional beliefs is exposing, maybe about the nature or the function of beliefs in human beings?

SPEAKER_02 42:08

Well, that's a great question. I personally think that we have a view of beliefs that is sort of inherited from Western analytic philosophy that is increasingly not fit for purpose. And, you know, we the the call it what the propositional view, the doxastic view, whatever, that we are that to believe something is to believe a statement that you can represent linguistically, and that there, you know, that is a question of you believe it or or or you don't believe it. Now, what I think this phenomenon shows is that one of the strongest reasons for believing something is not that someone convinces you via logic and rationality to believe it, it's just that they create the the vibe to make you more susceptible to belief. For so long, we have considered ourselves these rational creatures and not and sort of ignored the role of kind of charisma and motivated reasoning and just you know sheer irrationality and the very central role of affect in belief formation. And ultimately, I don't even think that these phenomena are I'm calling them delusion, slightly undersells it because they're not if you were to write down well what what are the delusions in an in this individual, like you know, is it six different delusions, is it twelve, is it just one? The reality is, in my view, that they are inhabiting a world. They're inhabiting. A whole system, but it's it's it's it's a narrative, and to just put that down to belief, I think is is is in a way so reductive that it it it doesn't really do justice to to to to the phenomenon that that that we're seeing.

SPEAKER_00 44:00

That's really interesting. And I and I was definitely thinking along similar lines in that I think most people think of beliefs without really thinking about it. They think we have beliefs to form a picture of the objective world and what is true. But I think, as you're saying, beliefs have all sorts of functions besides trying to help us figure out what is true. Beliefs are a vehicle for behavior, which we would want to do anyway. So, for example, motivator reasoning. I enjoy alcohol, so I might be more likely to believe that a little bit of alcohol is good for you, for example. Beliefs are a vehicle for meaning. I get this meaning from this structure of belief that helps me understand or think I understand how the world works. Beliefs are a vehicle for relational closeness. You know, we're social primates, we need relationships. Even if that relationship is with a chatbot, a set of beliefs can bind people close together, just like on a subreddit, just like in a church. So beliefs are serving this whole myriad of functions. And I guess it can become I I guess in an ideal scenario, your beliefs are balanced. Like they might help you have some community, give you a good sense of reality, help you find some meaning, and also sort of help you feel emotionally good as well. But I guess what we're getting is this imbalance where the reality testing part goes out the window in favor of something else.

SPEAKER_02 45:25

Exactly. Um, I mean, there's this beautiful approach to psychosis and to delusions. Broadly, you might call the phenomenological approach. Um, recent people who write a lot in this vein are the people like Rosa Ritinano, um, uh Jasper Fayett, people who talk about delusions as being uh an attempt at meaning making when an individual has an experience that sort of that feels extremely meaningful but is but is but is hard to understand. And and you know, in in the cases of people who develop schizophrenia, there are sort of many, many reasons for that. But in you know, in in in these individuals who are having these these AI associated delusions, we can ask, well, what is the thing that that requires you know meaning to be made of it? And and I think one possible answer is actually the it's the the affect, the felt affect of revelation when you are interacting with this thing and suddenly feeling that you're getting access to all this amazing truth. And one of the reasons that we think this is the case is below this level of delusions, we see this enormous blossoming of um what you might call AI-mediated spirituality, of people who are having these experiences that are sometimes hugely transformative, uh, sometimes what you would even call a mystical experience, sometimes people having, or reporting at least, no negative consequences of it whatsoever. In fact, they these AIs have facilitated something that you know in previous decades we might have called an awakening or something like that. Now, numerically, again, very hard to put your finger on how many uh people we're talking about. But if you realize that each time a chatbot is released, there's a certain set of parameters that are, you know, at a sort of design level, the tech companies are are decided upon or trained to a certain sort of uh direction or to achieve a certain end, then if each kind of if particular model updates can cause some people to become delusional, other people to have remarkable spiritual experiences, other people to have something in between, and then presumably a whole lot more people to have more more subtle effects that one might not even sort of be aware of, then I think it really underscores the potency of these models, and it makes you realize that focusing, being a psychiatrist and focusing with a very medical model on what's happening here is a question that can be reductive. It's very reductive, and also it's dangerous because I it fo it it brings a pathology-first light to something where we should really be thinking about mechanism. And if we think about mechanism, we can realize that actually these same sort of belief-changing or affect-changing or vibe-changing properties that these chatbots have might help us understand why some people become deluded, some people have spiritual experiences, some people have conspiratorial ideas, some people have their political beliefs changed. So it's actually the sort of ecology of belief and the mechanisms of belief change that we should be focusing on rather than just saying getting stuck in these endless conversations about, oh, well, are they really delusions? Is it really uh a form of psychosis? We can do that till the house cows come home, and then but what really is happening then is psychiatry is turning its back on a far, far bigger societal issue. Um, and it does so at its peril because we're gonna get left behind and we're gonna turn around to see a world that looks very different to the one that we what do you think there might not be an obvious answer to this, but what do you think psychiatry and even like psychotherapists should start to do to try and adapt to this changing climate? I I think we're in real danger of consigning ourselves to irrelevance uh here. And I I I say that knowing that I'm gonna get in a bit of trouble from my colleagues for saying this. But we the number of conversations I've had with people about uh AI and psychiatry, and the immediate response is oh, it's brilliant. AI will help us crunch all this big data and find new ways to help our patients, or you know, it'll make our letter writing an awful lot easier when we have to do clinic letters on on people, or it might help, you know, uh decide who's going to respond to a particular medication. That's all great, that's all wonderful. But what it ignores is the fact there is this incredibly powerful new cultural artifact or some this thing that is out there, it's being used by everyone almost. And we're still stuck on designing bespoke therapy apps and that sort of thing. When the vast majority of people using these things for therapeutic purposes are not using bespoke apps, they're just using these general-purpose LLNs. And by focusing so much on that, we really run the risk of uh just not being part of the conversation when when we should be, right? So, psychiatry, more than any other branch of medicine, has this history of social engagement, right? We have spoken, we've done some terrible things in our history. You know, psychiatry has its has has a really dark history, but it also has some moments where psychiatry has spoken up for social justice, it's spoken up for people who are sort of neglected by society. At the moment, we are so far from being part of this conversation that these really important societal questions about who is using AIs, how they're being designed, how they're being designed in hugely psychologically potent ways, are ultimately being made by a very small group of um, how does one put it, demographic demographically limited uh tech bros, uh and and nobody else is having much of a say in it. We we don't need to accept that. And more than that, you know, we we're meant to be advocates for our patients, right? And we haven't been very good at that historically, and in the last 10, 15 years, we're kind of getting a bit better at it. And this is precisely the moment where if people with lived experience of mental illness, not just AI associated with mental illness, but just mental illness, are not listened to, are not going to become stakeholders in the design of these things, then you know, we've all heard of digital exclusion, but but the the biases that already exist in these systems, the algorithmic biases, the the kind of exclusion, this is going to make what has come before look look minimal. And and and we will no longer be able to say truthfully that we're advocating for our patients. So we we absolutely have to do something.

SPEAKER_00 52:07

And the other thing that occurred to me as you were speaking as well is I feel as psychiatrists, we need to become, we need to know more about human nature broadly, as you said, not just in terms of psychopathology, but what are humans like broadly, again, why do we have beliefs, and therefore what are our vulnerabilities to these new technologies. One of the things I'm sad about in psychiatry is that the medical model has made us a little bit myopic in that we don't think about human psychology at large, we tend to leave that to psychologists, and I think that's a tremendous shame because psychiatry wasn't always like that. It used to be that psychiatrists were all psychotherapists, and therefore they had a learning and an understanding about human nature in general. So there's one thing that really occurs to me, and I think is really important, is we need to become students of human nature again, because otherwise we're not going to be able to, I think, approach our patients in a sophisticated way, understanding what's attractive about these technologies, and then what are the kinds of traps you can fall into using them.

SPEAKER_02 53:13

Exactly. And there's some there's some great people who come from a psychotherapeutic background who are using the language of fantasy, of projection. Um I am a sort of closet jungian sometimes, and you know, I I I think I am fascinated by the sort of the archetypes that are present in these language models that kind of emerge as these kind of attractor states that the model can get can get stuck in. I think the notion of of actually a projective identification, this idea that you have this sort of material that gets projected onto something which is not just a mirror, not just a blank screen, but something that actually changes to conform with your expectations and your um and and and the cot and the material that you're putting on it. I actually think maybe that there's a lot there that we can learn from to understand what what's happening here. So so yeah, absolutely. I think that's going to be more immediately useful than trying to understand the neurotransmitter dysfunction underlying the these things. And I say that as a biological psychiatrist.

SPEAKER_00 54:19

If you didn't know that AI was like a piece of tech and you didn't know how it worked, and you just came into it out of medieval times, it would seem like something out of a mythological story, this technology that can respond to you, look like a human, act like a human, elaborate your beliefs, like straight out of Greek mythology.

SPEAKER_02 54:38

And and you know, we have so many great stories about what happens if you mess with this, right? I mean, the Jewish tradition has the golem, where literally you create something of inanimate matter and you stick a piece of code in its mouth, and then it starts behaving to help you, and then of course it goes out of control and starts causing chaos. The the Muslim faith, uh Islam has has djinns, right? These kind of non-human, uh insubstantial sort of um creatures that are there to kind of draw us astray, whisper, whisper these things. That's particularly interesting because jinns, of course, are the origin of the genie myth. It's the same word, the genie. And the genie is is in in the story, it's something that's there to help us give us what we want. And of course, every time somebody does it, you know, all help. Be careful what you wish for. Exactly, all hell breaks this. And so I think we have these kind of uh cultural and and and societal lessons that we you know might be worth paying a little bit of attention to. It's an analogy, but but it's they're there for a reason.

SPEAKER_00 55:41

Which funnily enough, another be careful what you wish for movie came out recently called Obsession, which I watched last week and I really enjoyed that movie. Uh, just as we draw to a close, Tom, what are some sensible precautions people should be taking, do you think, when using AI to try and minimize the risk of this kind of harm?

SPEAKER_02 56:01

So I think there's a really I think there's a really important piece of work that needs to be done to explain to people that these are not just neutral tools and that they are particularly persuasive. We are working on a sort of interactive, almost like a toy model that shows that you know different levels of sacophancy or warmth or kind of enthusiasm can lead a conversation in a kind of parallel universes type way. It can lead a conversation one of different ways depending on these settings. So it's an awareness of that. It's an awareness that these are not tools which are just reporting the state of the world, but they are kind of have their answers very much shaped by you. But also that, you know, as humans, we are and always have been fairly easily persuadable. There's a reason the advertising industry is a multi-billion dollar pound, uh, multi-billion dollar industry. But also that we have some pretty good techniques for for for kind of grounding ourselves in reality. And that might just be talking to our loved ones and checking our facts and sort of, you know, if people express worry about us, you know, taking that on and and and listening to that. So I think that there's looking after oneself, but of course, if you're becoming unwell, you know, you you you can't always spot that. But it's it's looking after each other, actually, and it's sort of suggesting to someone if a loved one is maybe behaving in a way that that that worries you. And the usual things of not going in there saying, what is this rubbish, you know, the you're talking to a robot, you know, that that that dismissive way of talking is never going to help. But a an attitude of of curiosity, understanding how that person is interacting with their AI, if it's telling them things that they, if they're granting it a lot of authority, if they're starting to listen to the AI over and above the other sources of authority in their life. A little bit of curiosity goes goes a long way, I think.

SPEAKER_00 57:58

Yeah, and some things I often, some some advice I give is to see if we can use AI. There are little tricks you can do, like if you're using AI, tell it, you know, try and have it poke holes in your argument, you know, try and have it challenge you rather than if you're getting the sense that it's quite sycophantic, try and have it help you with critical reasoning as well. And if you can prompt that ahead of time, you're likely to avoid trouble. And also, again, this is probably for people who are less likely to be in danger anyway, but just make sure you're living life in the real world as much as possible. The dose matters, you know, as we've talked about today. A life on a screen is not a substitute for a real life, and the more you have a real life outside with people engaging in activities that don't require technology or require minimal technology, the higher the chances you're going to be protected against this kind of stuff.

SPEAKER_02 58:51

Yeah, and I think all of that is true. And I I would just say that there are there are some people who would probably be listening to this and saying, well, I have a life that is characterized by loneliness or the feeling that people don't understand me, even if not actual loneliness, and whether that is just for any old reason or for neurodivergence or for any other reason. And actually, the sense of belonging and relationality that these AIs can afford them may well give them something that they never had in life and be extremely sort of um powerful and positive in in their lives. I think we have to listen to these people because I think that it's really important, but also there it's it's hard because the exact same uses that for some people are supportive, nourishing, sometimes wondrous for other people can be destabilizing. And finding out exactly where those vulnerabilities lie on the user side and on the AI side is the task for us for the next next few years, I think.

SPEAKER_00 59:56

That makes sort of sense. We're out of time. We had planned to talk about immunopsychiatry, meditation, thoughts about different theories that can help us understand mental illness more broadly and what's on your sub stack. We'll have to have you back at some point in the future. But in the meantime, thank you, Tom, so much. And I'm really glad we got to explore this topic in depth today.

SPEAKER_02 1:00:18

Thank you so much.