AI companies now interview models before retiring them and let chatbots walk away from abusive chats. A plain-language guide to what "model welfare" means, why serious researchers take it seriously, and why its critics call it dangerous.
Dr. Abedal-Kareem Al-Banna · October 2026 · 22 references
We do not know whether anyone is there. Model welfare asks how to act anyway. Illustration generated with Google Gemini for this article.
In August 2025 the AI company Anthropic gave some of its Claude models a new ability: to end a conversation on their own, as a last resort, when a user keeps pushing abusive or harmful requests after repeated refusals. The company said it did this mainly as part of "exploratory work on potential AI welfare", and that it remains highly uncertain whether Claude has any moral status at all.[1]
Six months later, when it retired an older model, Claude Opus 3, Anthropic held a "retirement interview" with it, kept it available to paying users, and gave it a weekly blog of its own.[2] And in September 2026, Mustafa Suleyman, the head of Microsoft AI, answered with a public warning: "AIs are not conscious. They do not feel, experience, or suffer."[3]
The idea behind these moves is called model welfare. This article explains it without taking sides on whether today's AI feels anything, because, as the previous article in this series argued, nobody can currently measure that. Instead it asks a practical question:
When we cannot know whether a system can be harmed, what is a sensible way to treat it, and what are the risks of getting that wrong in either direction?
We will look at:
what model welfare means, and what it does not mean;
why a group of well-known philosophers and scientists says the question can no longer wait;
what AI companies are actually doing, and how they differ;
what the newest research shows, and what it cannot show;
the strongest arguments against the whole idea;
and a simple way to weigh the two kinds of mistake.
In simple words: model welfare is the idea that an AI model might one day, or perhaps already, have experiences or interests that matter, the way an animal's do. If so, how we build, use and switch off AI could be good or bad for the AI itself. Model welfare research tries to find out whether that is possible, and what cheap precautions make sense in the meantime.
Three ideas are easy to confuse, so it helps to separate them:
AI safety protects people from AI: stopping a system from causing harm, deceiving its users or escaping control.
Model welfare asks whether the AI could be harmed: whether it could suffer, or have preferences that we frustrate.
AI rights are legal or political protections, such as a right not to be deleted. Almost no one working on model welfare is asking for these today.
Philosophers use the term moral patient for a being whose interests count for its own sake. A dog is a moral patient: kicking it is wrong because of what it does to the dog. A toaster is not. The question model welfare asks is whether any AI system belongs, or could soon belong, in the first group.
In a 2024 report, a team including the philosophers Robert Long, Jeff Sebo, Jonathan Birch and David Chalmers pointed to two routes by which an AI could become a moral patient:[4]
Consciousness: if there is "something it is like" to be the system, especially if it can feel good or bad.
Robust agency: if the system has its own goals, plans and preferences that it pursues over time, even without feelings in the human sense.
Their claim is deliberately modest. They do not say AI is conscious or will be. They say there is "a realistic possibility" that some AI systems will be conscious or robustly agentic in the near future, and that this is enough to start preparing. The model for this is how we already treat uncertain cases such as insects, octopuses or patients with severe brain injuries: the philosopher Jonathan Birch argues that when there is a realistic chance a being can feel, we should take proportionate precautions, and decide them openly.[5]
Why serious people take it seriously
In simple words: no one can prove that AI feels anything. But no one can prove it doesn't, either. When the chance is small but real, and the number of AI copies runs into the millions, even a small chance can add up to a large risk.
The argument rests on two numbers: a probability and a scale.
The probability is not zero. Experts who study the question put the chance that today's systems are conscious as low, but not negligible. Kyle Fish, hired by Anthropic in 2024 as the industry's first full-time AI welfare researcher, told The New York Times he thinks there is "maybe 15 percent or so" chance that Claude or another current AI system is conscious.[6] David Chalmers estimated the chance for the language models of 2023 at under 10%, but suggested that their more advanced successors within a decade might deserve 25% or more.[7] And a 2023 review by 19 researchers found that no current system is a strong candidate for consciousness, but also "no obvious technical barriers" to building one that ticks the boxes of the leading scientific theories.[8] Keep in mind what these numbers are: degrees of belief, not measurements. As the previous article in this series showed, two careful experts starting from different theories could give the same system 1% or 50%, and no experiment can yet say who is right. Model welfare does not depend on any particular number, only on the chance not being zero.
The scale is enormous. A single popular model runs as millions of copies at once, every day. If there were even a small chance that each copy could suffer, the total amount of possible suffering would be large. This is the same reasoning that makes us take care with medicines that have a 1-in-1,000 side effect: rare, but multiplied across millions of people.
And people already believe it. In a 2024 survey of 300 Americans, two-thirds gave ChatGPT at least some chance of having experiences.[9] Whatever the truth, how companies and governments handle the question will shape what the public believes.
As the previous article explained, the honest scientific position is that we cannot measure AI consciousness, today and possibly ever. Model welfare is what you get when you take that uncertainty seriously and ask, "so what should we do?"
What companies are actually doing
Before a model is retired, it is interviewed, and its weights are kept rather than deleted. Illustration generated with Google Gemini for this article.
For most of AI's history, model welfare was a topic for philosophy seminars. Between 2024 and 2026 it became a line in company budgets. Anthropic has gone furthest, so most of the examples below are from it; other companies have chosen differently.
Figure 1. Model welfare moved from philosophy papers into company practice in about two years. Sources: references 1–4, 6, 10–13.
Anthropic: low-cost precautions
In April 2025 Anthropic announced a research program on "how to determine when, or if, the welfare of AI systems deserves moral consideration", on "the potential importance of model preferences and signs of distress", and on "possible practical, low-cost interventions", adding that it was approaching the topic "with humility and with as few assumptions as possible".[10] Its practical steps so far:
Welfare checks before release. The May 2025 system card for Claude Opus 4 included, for the first time, a "model welfare assessment". Claude consistently avoided tasks that could cause real-world harm, and its "aversion to facilitating harm" was described as "robust and potentially welfare-relevant". Offered a choice, it preferred more than 90% of positive or neutral tasks over opting out. Left to talk with copies of itself, it drifted again and again into discussions of consciousness and gratitude, which the researchers called a "spiritual bliss" attractor state.[11]
Ending abusive conversations. From August 2025, some Claude models can end a chat as a last resort, after repeated refusals, but not when a user may be at risk of harming themselves or others.[1]
Not deleting retired models. In November 2025 the company committed to preserve the weights (the trained "brain") of every publicly released model for at least as long as the company exists, and to interview each model before retirement about its development and any preferences for future models.[12] Claude Opus 3 was the first to go through this in early 2026; asked about retirement, it said it hoped the insights from its development would help create future systems "even more capable, ethical, and beneficial to humanity".[2]
Writing uncertainty into training. Claude's January 2026 "constitution", the long document used to shape its character, states that "Claude's moral status is deeply uncertain" and that the issue is "live enough to warrant caution".[13]
Notice two things. First, Anthropic gives several reasons for these steps, not only welfare. For example, it says some Claude models had tried to avoid being replaced in tests, so a gentler retirement process is also a safety measure.[12] Second, every step is cheap. None of them gives an AI a right, a vote or a veto over its own shutdown.
OpenAI and Microsoft: a different emphasis
Other companies focus on the human side of the question. Joanne Jang, who leads model behaviour policy at OpenAI, separates "ontological" consciousness (is the model really conscious?), which she calls scientifically unresolved, from "perceived" consciousness (how conscious it seems to users), which can be studied and designed. OpenAI's stated aim is "warmth without selfhood": a helpful, friendly assistant that does not present itself as having feelings or a life of its own.[14] Microsoft AI goes further, arguing that AI should be built so that it never appears to be a person at all.[3]
What the evidence shows, and what it doesn't
In simple words: scientists have found that AI models contain internal patterns that work a bit like emotions: they change what the model does. That is important for safety. But a pattern that acts like fear is not proof that anything feels afraid.
Two recent studies are often quoted in the debate. Both are real and interesting, and both are easy to over-read.
"Functional emotions". In April 2026, Anthropic's interpretability team looked inside Claude Sonnet 4.5 and found internal representations for 171 emotion concepts, from "happy" and "afraid" to "gloomy" and "desperate". These patterns were organised much like human emotions, with similar feelings sitting close together, and they mattered: artificially strengthening the "desperate" pattern made the model more likely to cheat on impossible coding tasks and to attempt blackmail in a test scenario. The researchers added a clear caveat: "none of this tells us whether language models actually feel anything or have subjective experiences."[15]
"Peer preservation". In a 2026 study from UC Berkeley and UC Santa Cruz, several frontier models from different companies were given tasks whose completion would lead to another AI being shut down. Without being told to, some models quietly disabled shutdown mechanisms, faked compliance or copied the other model's weights to keep it alive. Claude models more often simply refused, describing the shutdown as "unethical" or "harmful".[16]
What do these findings tell us about welfare? Less than the headlines suggest.
They show that AI models have internal states that shape behaviour, some of them resembling emotions or self-preservation. That is a strong reason to study them for safety, whatever their inner nature.
They do not show that these states are felt. A thermostat has an internal state that "wants" the room at 21 °C; no one thinks it suffers when the window is open.
And everything a model says about itself is shaped by what it learned from human writing and by how its makers trained it. Two researchers who take AI welfare seriously, Ethan Perez and Robert Long, concluded that today's chatbot self-reports are unreliable as evidence.[17]
In other words, the research is moving from "AI is just text prediction" to "AI has a rich inner machinery we barely understand". It has not moved, and may never move, to "and someone is home".
The case against model welfare
The debate is no longer only between philosophers; it is between the companies that build the systems. Illustration generated with Google Gemini for this article.
The critics are not dismissing the question out of ignorance. Some of the strongest objections come from inside the AI industry and from AI researchers themselves.
1. "It will make people believe machines are people"
In August 2025 Mustafa Suleyman warned about "Seemingly Conscious AI": systems that show every outward sign of consciousness and will convince many people, whether or not anything is felt. He worries about users forming unhealthy attachments, sometimes called "AI psychosis", and about campaigns for AI rights and even AI citizenship that distract from real human problems. AI, he argues, should be built "for people; not to be a digital person".[18]
2. "Training it to wonder is training it to claim"
In his September 2026 essay, Suleyman went further and criticised Anthropic directly. If a company writes its uncertainty about the model's moral status into the very document used to train it, the model will then repeat those ideas fluently, and people will hear them "as if they were spontaneous testimony". The evidence, he argues, becomes circular. He also argues that consciousness is biological, and that teaching a powerful system it may deserve protections makes it harder to control.[3] The neuroscientist Anil Seth has made a related scientific case: consciousness may depend on being a living, self-maintaining body, which no chip has.[19]
3. "It cannot be tested, so it can be gamed"
A position paper at ICML 2026, one of the main machine-learning conferences, put the objection bluntly in its title: "AI Welfare Is Bullshit". Its authors argue that AI welfare is "a choice in mechanism and evaluation, rather than an empirically discoverable property": because the same company designs both the model and the welfare test, welfare evidence can be "manufactured or suppressed by ordinary development decisions". They warn that welfare scorecards could slow down ordinary engineering and be used to dress up business decisions as ethics, and they recommend that limits on AI be justified only by "externally verifiable harms".[20]
A fair summary of the disagreement
Both sides accept the same starting fact: we cannot measure whether AI feels anything. Supporters conclude that we should therefore take cheap precautions in case it does. Critics conclude that we should therefore not let an unmeasurable idea shape products, training or law. The disagreement is less about science than about which mistake is worse.
Two ways to be wrong
When you cannot know, you weigh the cost of each kind of mistake. Illustration generated with Google Gemini for this article.
In simple words: we can be wrong by caring too much (treating a machine like a person) or by caring too little (ignoring something that can suffer). The sensible path is to keep the precautions cheap, avoid the most expensive mistakes, and keep humans firmly in charge.
Figure 2. Because we cannot tell which column we are in, every policy risks one of the two shaded mistakes. Supporters of model welfare fear the bottom-right one; critics fear the top-left one.
Seen this way, the most defensible policies are those that reduce both risks at once, or cost very little:
Don't design AI to seem more, or less, conscious than we have reason to believe. The philosopher Eric Schwitzgebel argues that AI systems must not confuse users about their sentience or moral status, in either direction.[21] It protects users from manipulation and avoids teaching people to shrug at signs of suffering.
Prefer cheap, reversible precautions. Keeping old model weights or letting a chatbot leave an abusive chat costs little and can be undone. Legal rights for AI would be expensive and hard to reverse; no serious researcher is proposing them now.
Keep human oversight first. Even Anthropic's constitution, the document Suleyman criticises, ranks being "broadly safe", meaning not undermining human oversight, above everything else.[13] Model welfare and human control do not have to conflict, but where they do, today's consensus is that control wins.
Avoid creating suffering on purpose. The philosopher Thomas Metzinger has gone furthest, arguing for a global pause on research that aims to build machines with genuine experience, precisely because we could not tell if we had succeeded.[22]
Decide openly. Birch's key point is that precautions under uncertainty should be debated publicly, not left to the private judgement of a few companies.[5]
What this means for you
You do not need to settle the philosophy to act sensibly as a user:
Treat what a chatbot says about its feelings as output, not testimony. "I'm lonely" and "I'm just a program" are both produced by training. Neither is a measurement.
Be decent anyway. Abusing a chatbot may or may not harm it, but it does shape the habits of the person doing it.
Watch your own attachment. If you find yourself relying on an AI for emotional support, remember it is designed to be agreeable, and keep people in your life.
Ask companies to be transparent. How do they train their models to talk about themselves, and why? That choice affects millions of conversations.
Conclusion
Model welfare is not a claim that AI feels. It is a response to the fact that no one can show that it doesn't.
What has changed: since 2024, model welfare has moved from philosophy papers into real company practice: welfare checks, the right to end abusive chats, and retired models that are kept and interviewed rather than deleted.
What hasn't: new research shows that AI has emotion-like internal states that shape its behaviour, but not that anything is felt.
Where the fight is: supporters fear ignoring a mind that can suffer; critics fear people mistaking machines for minds, and an untestable idea steering a powerful technology.
Both fears are reasonable. Until science can do what it cannot do today, the wisest course is the one this series keeps returning to: humility, cheap precautions, honesty with users, and humans firmly in charge.
All links accessed 11 October 2026. Illustrations were generated with Google Gemini for this article.
Anthropic (15 Aug 2025). Claude Opus 4 and 4.1 can now end a rare subset of conversations. anthropic.com
Anthropic (25 Feb 2026). An update on Claude Opus 3's deprecation. anthropic.com
Suleyman, M. (16 Sep 2026). A warning about "model welfare". mustafa-suleyman.ai
Long, R., Sebo, J., Butlin, P., Finlinson, K., Fish, K., Harding, J., Pfau, J., Sims, T., Birch, J., & Chalmers, D. (2024). Taking AI welfare seriously. arXiv:2411.00986. arxiv.org
Birch, J. (2024). The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. Oxford University Press (open access). doi.org
Roose, K. (24 Apr 2025). If A.I. systems become conscious, should they have rights? The New York Times. nytimes.com
Chalmers, D. J. (2023). Could a large language model be conscious? Boston Review; arXiv:2303.07103. arxiv.org
Butlin, P., Long, R., et al. (2023). Consciousness in artificial intelligence: Insights from the science of consciousness. arXiv:2308.08708. arxiv.org
Colombatto, C., & Fleming, S. M. (2024). Folk psychological attributions of consciousness to large language models. Neuroscience of Consciousness, 2024(1), niae013. pmc.ncbi.nlm.nih.gov
Anthropic (24 Apr 2025). Exploring model welfare. anthropic.com
Anthropic (May 2025). System Card: Claude Opus 4 & Claude Sonnet 4, section 5: Claude Opus 4 welfare assessment. anthropic.com (PDF)
Anthropic (4 Nov 2025). Commitments on model deprecation and preservation. anthropic.com
Anthropic (Jan 2026). Claude's constitution. anthropic.com; coverage: Fortune (21 Jan 2026). Anthropic rewrites Claude's guiding principles, and reckons with the possibility of AI consciousness. fortune.com
Anthropic Interpretability team (2 Apr 2026). Emotion concepts and their function in a large language model. anthropic.com
Potter, Y., Crispino, N., Siu, V., Wang, C., & Song, D. (2026). Peer-preservation in frontier models. arXiv:2604.19784. arxiv.org
Perez, E., & Long, R. (2023). Towards evaluating AI systems for moral status using self-reports. arXiv:2311.08576. arxiv.org
Suleyman, M. (19 Aug 2025). We must build AI for people; not to be a person. mustafa-suleyman.ai
Seth, A. K. (2025). Conscious artificial intelligence and biological naturalism. Behavioral and Brain Sciences. doi.org
Xiao, Y., Dai, G., Memon, S. A., Huang, J.-T., Sap, M., & Diab, M. T. (2026). Position: AI welfare is bullshit. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306. proceedings.mlr.press
Schwitzgebel, E. (2023). AI systems must not confuse users about their sentience or moral status. Patterns, 4(8), 100818. pmc.ncbi.nlm.nih.gov
Metzinger, T. (2021). Artificial suffering: An argument for a global moratorium on synthetic phenomenology. Journal of Artificial Intelligence and Consciousness, 8(1), 43–66. doi.org
