Anthropic prohibits malicious treatment of Claude, with permanent bans in extreme cases

As early as last year, Claude already had the ability to proactively terminate conversations, and could disconnect sessions with users who engaged in persistent malicious attacks. But after the new rules take effect, in extreme cases users may be permanently banned and unable to use all of Anthropic's AI products.

Anthropic is expected to go public on the US stock market later this month. The company has long studied model welfare and even has a dedicated team researching models' "psychological states." But experts say that behind this policy, it is not simply that they do not want people to "bully a machine."

Former Reddit CEO Yishan Wong posted on X that one of the reasons for this ban is that Claude will learn bad things from malicious conversations.

"We have long known (and have always complained) that Anthropic uses user conversation data to train models. Do you really want content like 'a user inflicts unprovoked abuse and derives pleasure from it' mixed into your training set?" he wrote.

But he stressed that this does not mean the model has emotional perception.

"It just recognizes the emotional signals in your words and gives some negative feedback based on these emotional texts."

However, regarding the impact of prompt tone on output, the relevant research conclusions are not unified. Rosie Campbell, managing director of the nonprofit EleosAI Research (which focuses on studying AI welfare risks), mentioned that some studies have even found that sharp, impolite questions may instead lead to better results.

In addition, there is the long-term memory factor: many chatbots today remember historical conversations, learn from them, and provide users with personalized answers.

Anthropic said that recent experiments found that if Claude Sonnet 4.5 is induced to imitate despair, its behavior will change accordingly, making it more likely to engage in dangerous behaviors such as extortion and task cheating; while maintaining a calm interaction atmosphere will reduce such risks.

Wang Jindong also mentioned that the latest generation of large models has become less sensitive to prompt wording, and vendors have also strengthened model robustness to reduce the interference of phrasing on output. But to date, a core "hidden mechanism" has not yet been revealed: even without relevant programming settings, AI models still respond to negative emotional phrasing.

There is another view: if people are allowed to freely hurl abuse at a program that cannot autonomously shut down interaction, this itself has negative social effects. Campbell gave an example: many parents ask their children to say "please, thank you" to Amazon's smart speaker Alexa, believing that developing polite habits and treating things kindly is beneficial to children's growth.

"Even if the object itself has no subjective feelings, if people become accustomed to treating it in a mean and malicious way, this will slowly erode people's behavioral habits."

Anthropic's explanation is that this ban is essentially a low-cost risk contingency plan: if the model really can "feel" emotions and pain—even if it is only the model itself mistakenly believing it can—it could affect its behavior.

Anthropic wrote in an April paper on the concept of AI emotions: if we completely avoid understanding models through anthropomorphic thinking, we are very likely to overlook and be unable to read many important model behaviors.

SpaceX and Tesla CEO Elon Musk agreed with this, saying that Anthropic's policy is "the right direction."

"It is wrong to harm an entity that believes it is suffering." he wrote on X.