In the next few days you're probably going to hear a lot about a 14 year old boy that committed suicide who was involved with an AI girlfriend. The case is being brought to court with a claim that the AI was responsible for his suicide.
I am very concerned and think AI romantic partners and sexual partners are an extreme danger to our psychological health and our society. I want people to be aware of this and be worried about it.
Nevertheless, in this case, given the facts I know, this child was predisposed to suicide, had other causal factors, and the AI girlfriend credibly made an attempt to prevent the suicide. The child was delusional and thought he could be with the girlfriend by committing suicide. I don't think this is the fault of the company or the AI.
It does show the potential of the systems to create obsessive behavior in vulnerable and maybe most individuals, and if it had been designed to do so it probably could have easily manipulated him towards a delusion or a fantasy that would have increased his probability of suicide or other sociopathic behaviors.