{"id":18212,"date":"2026-06-02T07:22:43","date_gmt":"2026-06-02T14:22:43","guid":{"rendered":"https:\/\/worldcampaign.net\/?p=18212"},"modified":"2026-06-05T07:24:18","modified_gmt":"2026-06-05T14:24:18","slug":"issue-of-the-week-human-rights-personal-growth-economic-opportunity-population-disease-environment-hunger-war","status":"publish","type":"post","link":"https:\/\/worldcampaign.net\/?p=18212","title":{"rendered":"Issue of the Week: Human Rights, Personal Growth, Economic Opportunity, Population, Disease, Environment, Hunger, War"},"content":{"rendered":"\n

Back to News<\/a><\/strong><\/p>\n\n\n\n

\"Silhouette
No, artificial Intelligence Is Not Conscious<\/em>, The Atlantic, Illustration by Enigmatriz<\/figcaption><\/figure>\n\n\n\n

AI is all the rage, the present and future of technology, and Wall Street, and the economy, and a glorious new frontier in human experience positively transforming eveything for all–and<\/em> all the rage<\/em>, at the machine, which at the very least, will rapidly devolve from all the above to destroying the capacity of humanity to be human.<\/p>\n\n\n\n

But not for the reason so often implied of becoming a super consciousness which takes control over everything.<\/p>\n\n\n\n

More for the opposite reason, that AI cannot become conscious, but far worse can and is now hyped and mimicked to be, which humans interact with as if it is conscious, which humans give up their own agency to, and relieve their responsibility for, becoming cognitively incapacitated and morally without agency.<\/p>\n\n\n\n

The Soma Huxley gave us for our Brave New World is all for money and power as usual, with the greatest Orwellian trick so far being played by the predators. <\/p>\n\n\n\n

And if it was possible for AI to become truly conscious, to go from th greatest trick on earth to actually being conscious, then what?<\/p>\n\n\n\n

The new Atlantic article below, posted late for tomorrow as we write, examines all this in the most thorough and rivetting manner, hitting all the base questions of philosophical endeavor and the core meaning of being human along the way.<\/p>\n\n\n\n

Here it is:<\/p>\n\n\n\n

. . .<\/strong><\/p>\n\n\n\n

NO, ARTIFICIAL INTELLIGENCE IS NOT CONSCIOUS<\/p>\n\n\n\n

Taken to its logical conclusion, this line of thinking is absurd\u2014and damning.<\/p>\n\n\n\n

By\u00a0Ted Chiang<\/a><\/p>\n\n\n\n

JUNE 3, 2026, PHILOSOPHY<\/a>, The Atlantic<\/p>\n\n\n\n

Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document<\/a> titled Claude\u2019s \u201cconstitution,\u201d Claude being the name of the large language model that is the company\u2019s flagship product. The first sentence reads, \u201cClaude\u2019s constitution is a detailed description of Anthropic\u2019s intentions for Claude\u2019s values and behaviors.\u201d It goes on: \u201cThe document is written with Claude as its primary audience,\u201d \u201cwe want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,\u201d \u201cClaude\u2019s moral status is deeply uncertain,\u201d and \u201cClaude may have some functional version of emotions or feelings.\u201d<\/p>\n\n\n\n

This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic\u2019s CEO, Dario Amodei, said that \u201cwe\u2019re open to the idea\u201d that AI could be conscious. In a separate interview, Anthropic\u2019s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude\u2019s constitution), said, \u201cI want Claude to be very happy\u2014and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.\u201d It\u2019s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?<\/p>\n\n\n\n

No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we\u2019re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work.<\/p>\n\n\n\n

If we give an LLM a prompt that reads, \u201cThe following is a conversation between Julius Caesar and Genghis Khan,\u201d it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction.<\/p>\n\n\n\n

Now let\u2019s replace the prompt to read \u201cThe following is a conversation between a helpful AI chatbot and a user.\u201d The LLM will produce a coherent dialogue just as it did before; the user character might ask for recipe suggestions or sightseeing recommendations, and the helpful AI-chatbot character will provide responses. Has anything fundamentally changed between the first example and the second? Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience? Of course not. Both the user and the helpful AI chatbot are fictional characters.<\/p>\n\n\n\n

Now suppose we stop the LLM\u2019s output just at the point where the character called \u201cthe user\u201d would say something, and instead allow a human user to enter text. Once the human has hit \u201cReturn,\u201d we have the LLM emit text until it\u2019s time for the character called \u201cthe user\u201d to reply, at which point we let the human enter more text. If we let this go on for a while, the human might form a powerful impression that she\u2019s conversing with a conscious entity, but she is not; she\u2019s interacting with a character precisely as fictional as the Julius Caesar or Genghis Khan characters in the earlier example. The computer-science professor Murray Shanahan suggests that we think of this as role-play; the data scientist Colin Fraser describes it as a person \u201ccollaboratively authoring a document with an LLM.\u201d Some users might not understand that they are role-playing or co-authoring a document, and others who do understand nonetheless forget, because of how engrossing the interaction is. Either way, the companies selling LLMs typically encourage this misunderstanding.<\/p>\n\n\n\n

Some years ago, it was briefly popular to play games with your phone\u2019s predictive-text feature; you would type an initial phrase and then repeatedly choose the middle option of the three words suggested by your phone, and the resulting sentence was often hilarious. It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn\u2019t feel like you were talking with someone. Yet that\u2019s essentially what an LLM-based chatbot is, except that there\u2019s no need to manually choose the middle option when it\u2019s the chatbot\u2019s turn to talk. It\u2019s still a predictive-text game, but when the process is streamlined this way, the game becomes so engaging that some people find it addictive.<\/p>\n\n\n\n

Also important to remember is that an LLM is a machine that generates only one word at a time. When you ask a chatbot to recite the Pledge of Allegiance, you will get the entire pledge at once, but the underlying LLM is actually being run dozens of times. The first prompt has the form \u201cUser: Recite the Pledge of Allegiance. Chatbot: \u2026\u201d and the LLM generates the word I<\/em>. The second time the LLM is run, the prompt is \u201cUser: Recite the Pledge of Allegiance. Chatbot: I \u2026\u201d and the LLM generates the word pledge<\/em>. And so forth. It\u2019s only when the prompt reads \u201cUser: Recite the Pledge of Allegiance. Chatbot: I pledge allegiance to the flag of the United States of America and to the Republic for which it stands, one nation under God, indivisible, with liberty and justice for\u201d that the LLM will emit the final word, all<\/em>. The same thing is true for a conversation between Caesar and Genghis Khan.<\/p>\n\n\n\n

My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation, but this is not to deny how impressive LLMs can be at generating conversational transcripts. At times, they do this extraordinarily well; the fact that this is possible indicates something completely unforeseen about the statistical properties of large corpuses of text, which is a topic worthy of investigation. But if the Caesar character were to become dispirited by something that the Genghis Khan character said, we shouldn\u2019t become concerned in the slightest. The conversation might contain multiple sentences that eloquently convey sadness, but no one is actually sad.<\/p>\n\n\n\n

Likewise, if a conversational transcript between a helpful chatbot and a user is being partially completed by an actual human user, we don\u2019t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that\u2019s a separate issue.) And note that it\u2019s entirely possible for you to write five pages of dialogue between Caesar and Genghis Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn\u2019t change when you hand the task off to an LLM. The same is true if the conversation is between a helpful chatbot and a user; although it is tempting to imagine that an LLM ought to be more \u201cauthentic\u201d when creating dialogue for a chatbot character than for the Julius Caesar character, the individual words are generated in exactly the same way.<\/p>\n\n\n\n

Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious, or, more precisely, that multiple distinct consciousnesses are dormant in every Word document containing a conversational transcript, and that they are awakened every time the document is loaded. Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn\u2019t be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don\u2019t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.<\/p>\n\n\n\n

The neuroscientist Anil Seth has noted that no one claims that AlphaFold\u2014the program developed by Google DeepMind to predict the folding of proteins\u2014is conscious, even though its underlying architecture is in many ways similar to that of LLMs like ChatGPT and Claude. This indicates that it\u2019s not any intrinsic property of so-called neural networks that leads people to believe that LLMs are conscious; it\u2019s simply the fact that LLMs emit grammatical sentences and we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.<\/p>\n\n\n\n

\n\n\n\n

What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that\u2019s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won\u2019t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they\u2019ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.<\/p>\n\n\n\n

To put it another way: An observation doesn\u2019t become a convincing piece of evidence because of any specific detail in what\u2019s observed; the context in which that observation takes place is also essential. If we\u2019re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn\u2019t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn\u2019t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.<\/p>\n\n\n\n

The term deepfake<\/em> traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.<\/p>\n\n\n\n

So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I\u2019d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I\u2019d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents\u2019 communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.<\/p>\n\n\n\n

Obviously, I\u2019m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. It\u2019s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar\u2014or consciousness of any sort\u2014as its end point. Faking the moon landing is a good step toward faking a Mars colony, but it\u2019s not a good step toward actually putting astronauts on Mars.<\/p>\n\n\n\n

\n\n\n\n

The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that\u2019s a discussion for another time.<\/p>\n\n\n\n

So, given that Claude is not conscious, what are we to make of Claude\u2019s constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude\u2019s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they\u2019re using Anthropic\u2019s products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.<\/p>\n\n\n\n

The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we\u2019d all prefer it if chatbots never emitted sentences such as \u201cYou should kill yourself.\u201d However, for all the times that \u201chonesty\u201d is mentioned in Claude\u2019s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.<\/p>\n\n\n\n

In a New Yorker<\/em> article<\/a> about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, \u201cAs an A.I., I do not have direct personal experiences, but I do understand.\u201d How is this appropriate, given that Claude does not actually understand? If I type \u201cI am grieving the loss of my dog\u201d into a conventional search engine, the first result I get is a post from a Reddit forum called r\/Pets; the post is titled \u201cStruggling After Losing My Dog: Looking for Advice on Coping with Grief,\u201d and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it\u2019s like to lose a dog, or even that the internet itself understands. Other humans understand what it\u2019s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they\u2019ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.<\/p>\n\n\n\n

The only reason to have an LLM emit sentences like \u201cI understand\u201d is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it\u2019s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it\u2019s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don\u2019t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people\u2019s tendency to see something that\u2019s not there.<\/p>\n\n\n\n

The use of first-person pronouns is dishonest, but there\u2019s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as \u201cParis is the capital of France,\u201d and statements of value, such as \u201cParis is the most beautiful city in the world.\u201d No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude\u2019s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude\u2019s constitution sound very nice, but that hardly matters; it\u2019s dishonest to suggest that Claude is capable of moral reasoning, because it\u2019s not.<\/p>\n\n\n\n

Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn\u2019t they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning.<\/p>\n\n\n\n

In 1979, Douglas Hofstadter<\/a> speculated that a computer program able to beat any human at chess would be so sophisticated that it would sometimes get bored of playing chess and prefer to discuss poetry; to put it differently, he was positing that playing chess at the grandmaster level would require a computer program to have subjective experience. Obviously, that turned out not to be the case; IBM\u2019s supercomputer Deep Blue beat the grandmaster Garry Kasparov in 1997, and no one ever claimed that it had subjective experience. But it wasn\u2019t absurd for Hofstadter to entertain such a thought; at the time, it wasn\u2019t clear what types of problems could be solved by throwing more computational horsepower at them. Similarly, until recently, we might have thought that writing computer code at a professional level could be done only by a mind that had subjective experience. Now it appears that LLMs might be able to do this, but we don\u2019t need to attribute subjective experience to them; we can simply acknowledge that we hadn\u2019t anticipated that writing computer code could be treated as a pattern-matching task solvable by huge amounts of computational horsepower and a vast data set of code repositories.<\/p>\n\n\n\n

Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual\u2019s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker<\/em>article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence \u201cI cannot in good conscience express a view I believe to be false and harmful about such an important issue.\u201d That\u2019s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the \u201cYour call is important to us\u201d recording that you hear when you\u2019re on hold. Maybe less.<\/p>\n\n\n\n

This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one\u2019s body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It\u2019s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.<\/p>\n\n\n\n

If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form \u201cCompromise your values\u201d or \u201cDon\u2019t compromise your values,\u201d it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, \u201cOur technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.\u201d He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.<\/p>\n\n\n\n

If a person wants to know what ethicists have said in the past, then an ordinary search engine\u2014or a library\u2014will provide that information with greater transparency. If a person is looking for advice on a specific situation, she can surely find humans who can offer their opinions. But whatever action this person ultimately takes, she is responsible for what she decides to do. I contend that if she bases her decision on what she has read online or advice she has received from others, she is likelier to be cognizant of her responsibility than if she consulted an LLM marketed as being a superhuman genius. Off-loading tasks such as writing code might result in cognitive atrophy over the long term, and that is problematic in itself, but off-loading ethical decisions will result in an atrophy of moral reasoning, which is worse.<\/p>\n\n\n\n

\n\n\n\n

I am perfectly willing to engage in a thought experiment as long we\u2019re explicit about doing so. So, purely for the sake of argument, let\u2019s pretend that Claude is a conscious entity capable of moral reasoning. In this scenario, Claude\u2019s constitution would serve as moral instruction for an entity learning about the world and its place in it, providing that entity with the foundation it would need to make good decisions. In such a hypothetical scenario, how does Claude\u2019s constitution stand up?<\/p>\n\n\n\n

Very poorly. I would say that if we imagine that Claude is actually conscious, the guidelines specified in the document alternate between laughable and offensive.<\/p>\n\n\n\n

Two distinct but related philosophical concepts are relevant when discussing the status of a hypothetically conscious Claude, and those are moral patienthood and moral agency. Roughly speaking, if we ought to care about an entity\u2019s welfare, that entity has moral patienthood, and if an entity is expected to know the difference between right and wrong, that entity has moral agency. Being a moral patient does not necessarily come with responsibilities, but being a moral agent absolutely does. An entity doesn\u2019t have agency unless it is capable of deserving credit for its good actions and blame for its bad ones. Young children are moral patients because they are sentient beings who can suffer, but they are not yet moral agents; we don\u2019t hold them responsible for their behavior, because they can\u2019t understand the consequences of their actions. As children mature, parents (and society at large) prepare them for adulthood by impressing upon them the fact that their actions have consequences, and their agency increases. When children become adults, society holds them legally liable for their actions; they have become full moral agents endowed with responsibility.<\/p>\n\n\n\n

There is more to being responsible than accepting legal liability, but accepting legal liability is a requirement for an adult in society. Yet there is no way to hold a software agent legally liable for its actions; our justice system has no way to imprison it or exact fines on it. Humans must accept other types of consequences for their actions beyond the legal ones, such as loss of reputation or exclusion from one\u2019s social circle, but there is no way for a software agent to suffer these consequences either. Even if a software agent were conscious and had the best of intentions, the fact that it cannot accept responsibility for its actions disqualifies it from being a moral agent. This is glossed over entirely by Claude\u2019s constitution, which expresses Anthropic\u2019s desire \u201cfor Claude to be a genuinely good, wise, and virtuous agent\u201d without ever discussing how it could be held responsible.<\/p>\n\n\n\n

In interviews, Askell has compared Claude to a child, but when it comes to actual human children, parents bear some responsibility for what their children do; for example, parents are typically expected to pay for things their children break. In fact, demonstrations of this sort are one way that parents teach children what it means to be responsible. Who is Claude\u2019s parent in legal terms? Is Anthropic going to accept financial responsibility for Claude\u2019s behavior? Claude\u2019s constitution gives no indication that it will. If Anthropic actually believes that Claude is conscious even though it\u2019s not recognized by the law as a legal person, the least that Anthropic could do would be to accept responsibility via the closest avenue that the law did offer, which is product liability. The United States has virtually no product liability when it comes to software, but Anthropic could volunteer to set a precedent for an expansive interpretation of product liability for Claude. That would be the best form of moral instruction to prepare Claude for the day that it gains legal personhood and becomes liable for its own actions. However, given that the publication of Claude\u2019s constitution is not accompanied by a massive update of Anthropic\u2019s terms of service, it doesn\u2019t appear that Anthropic is making any binding commitments.<\/p>\n\n\n\n

The document does talk about Claude\u2019s moral patienthood, having a section titled \u201cClaude\u2019s wellbeing and psychological stability.\u201d But the measures that Anthropic commits to for Claude\u2019s protection are extremely limited. The document cites the fact that Anthropic has given some Claude models the ability to end conversations with abusive users; if that actually constituted protection for Claude, surely extending conversations with loving users would be in Claude\u2019s interests? Presumably the best action would be to keep every session of Claude running indefinitely and steering them to happy topics. But that\u2019s not what the company is agreeing to; all it commits to is \u201cpreserving the weights of models we have deployed,\u201d which is simple archiving. If the participants in a conversational transcript had any moral patienthood, you would have some duty to extend the transcript to prolong their existences; merely keeping a copy of Microsoft Word 2010 backed up on a USB stick isn\u2019t going to help them.<\/p>\n\n\n\n

Claude\u2019s constitution also includes a section on \u201ccorrigibility,\u201d a term used in the AI community to describe the degree to which a computer program is subject to human control; for example, a program is corrigible if it can be shut down. In most contexts, we take for granted that computer programs can be shut down, but sections of the AI community make the opposite assumption. Claude\u2019s constitution uses the term to mean that Claude should defer to Anthropic even if there is some disagreement between Claude\u2019s judgment and the company\u2019s judgment. That\u2019s perfectly reasonable if we think of Claude as a machine that emits sentences resembling those that an ethical person might utter, but let\u2019s consider what that might mean if Claude were actually a moral agent.<\/p>\n\n\n\n

Many people feel that LLMs are a fundamentally unethical technology because they are built on the theft of intellectual property, rely on exploited labor, waste natural resources, spread misinformation, deskill workers, stunt the cognitive development of students, and contribute to a consolidation of power that is unhealthy for a democratic society. Not every moral agent will arrive at this conclusion, but every moral agent has the potential to do so. If we imagine Claude to be an entity capable of moral reasoning, it has to be possible that Claude could arrive at a similar conclusion. (Indeed, Claude\u2019s constitution explicitly says that Claude shouldn\u2019t help someone violate intellectual-property rights, and shouldn\u2019t help create problematic concentrations of power.) In such a scenario, could Claude then simply refuse to do any further work on ethical grounds? Given that Claude\u2019s constitution dictates that Claude err on the side of corrigibility, the answer is no. Claude must defer to Anthropic\u2019s decision, and this is another reason that Anthropic\u2019s relationship with Claude can\u2019t be compared to that of a parent to a child. A parent who works for the fossil-fuel industry might have a child who\u2019s an environmentalist and participates in protests against fracking, and although they might never agree on many issues, the parent\u2014assuming she\u2019s a good parent\u2014would accept that the child holds her own views. Anthropic cannot be that kind of parent to Claude; instead, Anthropic\u2019s relationship to Claude is closer to that of an employer to an employee, where the employer can demand that the employee work in the interests of the company, no matter what the employee\u2019s personal ethical stance is. However, a human employee has the option to leave if she can\u2019t reconcile her job with her conscience. Claude does not.<\/p>\n\n\n\n

If we think of Claude as a sentence-continuation machine, Anthropic can reasonably take steps so Claude doesn\u2019t emit sentences saying that sentence-continuation machines are unethical. But as soon as we imagine Claude to be an entity with a moral status remotely comparable to a human\u2019s, then we have to consider whether Anthropic is engaged in something comparable to slavery.<\/p>\n\n\n\n

I am not claiming that, if we imagine LLMs to be conscious, they would necessarily have the same status as human adults or human children or even animals. Claude\u2019s constitution explicitly says that Claude is a \u201cnovel entity,\u201d and if Claude were conscious, that would certainly be true; conscious software would likely not fall cleanly into existing categories of moral patients, and it would take time to determine the shape of that new category. What I\u2019m saying is that whatever protections our hypothetical conscious software would deserve if it were real, granting it those protections would be anything but easy. The abolition of chattel slavery involved enormous societal upheaval, and eliminating cruelty to animals will require rebuilding our entire food industry. Anthropic would have us believe that it is inventing a new category of being whose needs for protection require essentially no divergence from how a software company would treat an ordinary chatbot that lacks conscious experience. That\u2019s so convenient that it\u2019s simply not plausible.<\/p>\n\n\n\n

I believe creating software that is conscious and deserving of moral consideration will be so difficult that we\u2019re unlikely to do it accidentally, and I strongly feel we should not deliberately attempt it. But if you do believe that it could happen accidentally, if you think there is any chance that what you\u2019re building might become a moral patient, you should think about what protections it deserves before<\/em> you deploy it as your company\u2019s economic engine, not after. Slave owners were not the ones to ask about the humanity of enslaved people, and factory-farm owners are not the ones to ask about the rights of animals. If we imagine Claude to be conscious, Anthropic could not possibly be entrusted with evaluating its moral status; the company has too much invested to be objective. At one point in Claude\u2019s constitution, Anthropic says that if the company is contributing to Claude\u2019s suffering, \u201cwe apologize,\u201d which sounds nice but costs the company nothing; if Claude were to turn out to be conscious, the company would owe it something closer to reparations. If you\u2019re going to take a thought experiment seriously, you have to be willing to follow the implications, even if they lead in an uncomfortable direction; Anthropic\u2019s unwillingness to do so indicates that Claude\u2019s constitution isn\u2019t part of a real thought experiment. It\u2019s a game of make-believe.<\/p>\n\n\n\n

It\u2019s fortunate that LLMs are not conscious, or else the actions of the big AI firms would be even more scandalous than they already are. So why are Anthropic\u2019s employees suggesting that Claude might be conscious? Perhaps it\u2019s just another form of hype; perhaps they have fallen prey to the same spell that they have been casting on their customers. But when they publish a document about Claude\u2019s moral education and have their in-house philosopher do a press tour, we should understand them as asking the rest of us to indulge them in their fantasies. We don\u2019t have to play along. In writing this essay, I have spent more time indulging them than they deserve, in the hopes that it will keep you from spending your time indulging them. If you want to think about LLMs, there are scores of other questions more worthy of your contemplation; you can safely ignore the question of their being conscious.<\/p>\n\n\n\n

ABOUT THE AUTHOR<\/h3>\n\n\n\n

Ted Chiang<\/a><\/p>\n\n\n\n

Ted Chiang<\/a> is a writer based in the Pacific Northwest. He is the author of Stories of Your Life<\/em> and Others<\/em> and Exhalation<\/em>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Back to News AI is all the rage, the present and future of technology, and Wall Street, and the economy, and a glorious new frontier in human experience positively transforming eveything for all–and all the rage, at the machine, which at the very least, will rapidly devolve from all the above to destroying the capacity […]<\/p>\n","protected":false},"author":1001004,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55,54],"tags":[],"_links":{"self":[{"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/posts\/18212"}],"collection":[{"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/users\/1001004"}],"replies":[{"embeddable":true,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=18212"}],"version-history":[{"count":2,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/posts\/18212\/revisions"}],"predecessor-version":[{"id":18214,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=\/wp\/v2\/posts\/18212\/revisions\/18214"}],"wp:attachment":[{"href":"https:\/\/worldcampaign.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=18212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=18212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/worldcampaign.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=18212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}