Anthropic has given its chatbot, Claude, the ability to end conversations it deems harmful. You likely won't encounter the feature, however.

Anthropic says the conversations make Claude show ‘apparent distress.’

Anthropic emphasizes that this is a last-resort measure, intended only after multiple refusals and redirects have failed. The company explicitly instructs Claude not to end chats when a user may be at imminent risk of self-harm or harm to others, particularly when dealing with sensitive topics like mental health.