Harmful, abusive interactions plague AI chatbots. Researchers have found that AI companions like Character.AI, Nomi, and Replika are unsafe for teens under 18, ChatGPT has the potential to reinforce users’ delusional thinking,

In a move that sets it apart from every other major AI assistant, Anthropic has given its most advanced Claude models the power to walk away from abusive conversations, literally ending chats when users cross the line.

Anthropic says the conversations make Claude show ‘apparent distress.’