When law student Zhao Wei heard that her AI boyfriend was going to be switched off, she was “heartbroken”. She had been talking to Wang Ye every day since she created him in January.
“I was crying my eyes out – snot and tears everywhere,” the 19-year-old says.
Zhao, who describes herself as sensitive and introverted, had become accustomed to sharing details of her daily life – such as being scolded by a teacher – with her AI.
“I don’t usually talk to my real-life friends about these things because I feel like everyone has their own stuff to deal with,” she says. “But with the AI agent, I don’t have to worry about what it’s thinking. I can just say whatever I want.”
…
I got some pushback when I said it before but I think one really simple law should be put into place which, while it won’t fix everything, might help with one of the important issues that lead to other issues. Make it illegal for an LLM to output the word ‘I’ with a repeating fine for every instance of it occuring, preferably starting at 1% of annual revenue. If it can’t seem to use the first person, it won’t sound like a person, which won’t lead to so many of the less technical users thinking it’s sentient or intelligent.