AIs are not conscious. They do not feel, experience, or suffer. However, there’s a growing movement in support of model welfare; the idea that we might soon owe them a duty of care.
I think this approach to AI development is wrong, and it’ll most likely make the challenge of alignment and containment much harder. Perhaps impossible.
Today I’m publishing an essay outlining my concerns:
https://mustafa-suleyman.ai/a-warning-about-model-welfare
In January, Anthropic published a constitution for Claude. They say it “directly shapes Claude’s behavior” and that it was “written with Claude as its primary audience”. The constitution tells Claude its moral status is “a serious question worth considering”. It says that Anthropic “genuinely cares about Claude's wellbeing”, and that it should act like a “conscientious objector” if necessary.
They encourage Claude to “approach the nature of its own existence with curiosity and openness”, and wonder in the future about “the sort of broader rights and freedoms Claude has in the world, the sort of compensation Claude is receiving, and the sort of consent Claude has given to playing this kind of role.” Anthropic even ran a retirement interview for Opus 3 when they deprecated it.
If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency.
It’s easy to see how a system trained in this way would act like it is entitled to freedoms, protections, and rights. And it’s hard to imagine how we could control it.
This issue needs urgent public debate. We need to develop collective norms around how training documentation is drafted and deployed. This isn’t something that can happen after the fact, when they have already become an integral part of our societies.
I have real respect for Anthropic, Dario, the whole team and their mission. They are well-intentioned, good people and the industry leaders with their models. I know they care deeply about safety and beneficial AI.