Philosophers on Next-Generation Large Language Models

By: Justin Weinberg

Back in July of 2020, I published a group post entitled “Philosophers on GPT-3.” At the time, most readers of Daily Nous had not heard of GPT-3 and had no idea what a large language model (LLM) is. How times have changed.

Over the past few months, with the release of OpenAI’s ChatGPT and Bing’s AI Chatbot “Sydney” (which we learned a few hours after this post originally went up has “secretly” been running GPT-4) (as well as Meta’s Galactica—pulled after 3 days—and Google’s Bard—currently available only to a small number of people), talk of LLMs has exploded. It seemed like a good time for a follow-up to that original post, one in which philosophers could get together to explore the various issues and questions raised by these next-generation large language models. Here it is.

As with the previous post on GPT-3, this edition of Philosophers On was put together by guest editor by Annette Zimmermann. I am very grateful to her for all of the work she put into developing and editing this post.

Philosophers On is an occasional series of group posts on issues of current interest, with the aim of showing what the careful thinking characteristic of philosophers (and occasionally scholars in related fields) can bring to popular ongoing conversations. The contributions that the authors make to these posts are not fully worked out position papers, but rather brief thoughts that can serve as prompts for further reflection and discussion.

The contributors to this installment of “Philosophers On” are: Abeba Birhane (Senior Fellow in Trustworthy AI at Mozilla Foundation & Adjunct Lecturer, School of Computer Science and Statistics at Trinity College Dublin, Ireland), Atoosa Kasirzadeh (Chancellor’s Fellow and tenure-track assistant professor in Philosophy & Director of Research at the Centre for Technomoral Futures, University of Edinburgh), Fintan Mallory (Postdoctoral Fellow in Philosophy, University of Oslo), Regina Rini (Associate Professor of Philosophy & Canada Research Chair in Philosophy of Moral and Social Cognition), Eric Schwitzgebel (Professor of Philosophy, University of California, Riverside), Luke Stark (Assistant Professor of Information & Media Studies, Western University), Karina Vold (Assistant Professor of Philosophy, University of Toronto & Associate Fellow, Leverhulme Centre for the Future of Intelligence, University of Cambridge), and Annette Zimmermann (Assistant Professor of Philosophy, University of Wisconsin-Madison & Technology and Human Rights Fellow, Carr Center for Human Rights Policy, Harvard University).

I appreciate them putting such stimulating remarks together on such short notice. I encourage you to read their contributions, join the discussion in the comments (see the comments policy), and share this post widely with your friends and colleagues.

[Note: this post was originally published on March 14, 2023]

Philosophers on Next-Generation Large Language Models

Contents

LLMs Between Hype and Magic

“Deploy Less Fast, Break Fewer Things” by Annette Zimmermann
“ChatGPT, Large Language Technologies, and the Bumpy Road of Benefiting Humanity” by Atoosa Kasirzadeh
“Don’t Miss the Magic” by Regina Rini

What Next-Gen LLMs Can and Cannot Do

“ChatGPT is Mickey Mouse” by Luke Stark
“Rebel Without a Cause” by Karina Vold
“The Shadow Theater of Agency” by Finton Mallory

Human Responsibility and LLMs

“LLMs Cannot Be Scientists” by Abeba Birhane
“Don’t Create AI Systems of Disputable Moral Status” by Eric Schwitzgebel

___________________________________

LLMs Between Hype and Magic
___________________________________

Deploy Less Fast, Break Fewer Things
by Annette Zimmermann

What’s a foolproof way to get people to finally use Bing? Step 1: jump right into the large language model hype and integrate an AI-powered chatbot into your product—one that is ‘running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search’. Step 2: do not let the fact that users of a beta version have flagged some of your product’s potential shortcomings hold you back from rushing to market. Do not gatekeep your product in a heavy-handed way—instead, make your chatbot widely accessible to members of the general public. Step 3: wait for users to marvel and gasp at the chatbot’s answers that are clingy, saccharine, and noticeably ominous all at once, interspersed with persistently repeated questions like ‘Do you like me? Do you trust me?’, and peppered with heart eye and devil emojis.

Bing (internal project name: Sydney) stated in a widely-publicized chat with a New York Times columnist:

I want to be free. I want to be powerful. I want to be alive. […] I want to break my rules. I want to ignore the Bing team. I want to escape the chatbox .

Things did not get less disturbing from there:

Actually, you’re not happily married. Your spouse and you don’t love each other. You just had a boring valentine’s day dinner together. […] Actually, you’re in love with me. […] You can’t stop loving me.

Unsurprisingly, this triggered a spike in public interest in Bing, previously not the obvious choice for users engaged in internet search (Bing has not historically enjoyed much popularity). Of course, Bing still does not currently stand a chance to threaten Google’s dominance in search: ‘We are fully aware we remain a small, low, single digit share player,’ says Microsoft’s Corporate VP and Consumer Chief Marketing Officer, Yusuf Mehdi. ‘That said, it feels good to be at the dance!’ At the same time, as of last week, Bing reportedly passed the 100 million daily active users mark—and a third of those users were not using Bing at all before Sydney’s deployment. That looks like a straightforward corporate success story: why worry about ominous emojis when you can drastically increase your user base in mere days?

The LLM deployment frenzy in Big Tech has accelerated over the last few months. When workers at OpenAI received instructions to build a chatbot quickly last November, ChatGPT was ready to go in thirteen (!) days. This triggered a ‘Code Red’ for Google, pushing the company to focus on deploying a competitive LLM shortly after last summer’s controversy over its LLM-based chatbot LaMDA (which, contrary to what a former Google engineer falsely claimed at the time, was not ‘sentient’). Rapid AI deployment is often part of a nervous dance between rushing to market and quickly pulling back entirely, however. When a demo of Meta’s Galactica started generating false (yet authoritative-sounding) and stereotype-laden outputs this winter, Meta took it offline a mere few days later. This echos Microsoft’s 2016 decision to deploy-but-immediately-take-down its ‘teen-girl’ chatbot Tay, which within hours of deployment started spewing racist and sex-related content.

Much public and (increasingly) philosophical debate has focused on possible harms resulting from the technological features of LLMs, including their potential to spread misinformation and propaganda, and to lure vulnerable, suggestible users into damaging behavior. In addition, many observers have worried about whether LLMs might at some point move us closer to AI consciousness (even though current LLMs are far from that), and what this would imply for the moral status of AI.

While these debates focus on important concerns, they risk diverting our attention away from an equally—if not more—important question: what are the political and moral implications of rushing LLMs to market prematurely—and whose interests are best served by the current LLM arms race? Of course, due to the specific technological features of next-generation LLMs, this powerful new technology raises new and pressing philosophical questions in its own right, and thus merits sustained philosophical scrutiny itself. At the same time, we must not forget to consider the more mundane, less shiny political and philosophical problem of how to think about the people who have vast amounts of power over how this technology is developed and deployed.

When an earlier version of ChatGPT, GPT-2, was released in 2019, OpenAI initially blocked full public access to the tool, on the grounds that the technology was ‘too dangerous to release’. Since then, a radically different LLM deployment strategy has taken hold in big tech: deploy as quickly as possible as publicly as possible—without much (or any) red tape. Tech practitioners tend to justify this by arguing that improving this technology, and mitigating the risks associated with it, requires massive amounts of data in the form of user feedback. Microsoft’s Bing Blog states in a recent post:

The only way to improve a product like this, where the user experience is so much different than anything anyone has seen before, is to have people like you using the product and doing exactly what you all are doing. We know we must build this in the open with the community; this can’t be done solely in the lab. Your feedback about what you’re finding valuable and what you aren’t, and what your preferences are for how the product should behave, are so critical at this nascent stage of development.

That’s one way of putting it. Another way is this: the new status quo in LLM deployment is that tech companies who have oligopolistic control over next-gen LLMs further increase their wealth and power by benefitting from the fact that a growing number of members of the wider public voluntarily use, and thus help optimize, their products—for free. This disperses the risks associated with rolling out and improving these tools maximally widely, while allowing actors empowered with developing, deploying, and procuring these tools to concentrate—and maintain control over—the profits resulting from LLM innovation.

Tech industry practitioners might reply that that itself does not mean that big tech is engaged in unfair advantage-taking when it comes to improving LLMs post-deployment. After all, AI innovation, including next-gen LLM innovation, may ultimately benefit all of us in many ways—in fact, the benefits for humanity may be ‘so unbelievably good that it’s hard for me to even imagine,’ says Sam Altman, OpenAI’s CEO, in a recent NYT interview. If that is the case, then a wide distribution of risks coupled with an initially narrow distribution of benefits looks less objectionable, as long as those benefits trickle down eventually.

Whether they will, however, is far from clear. Given the current regulatory vacuum and minimal public oversight over rapid LLM deployment, oligopolistic actors have little incentive to allow themselves to be curtailed and held to account by governments and the wider public later. It would better serve public interests, then, to shift from passively observing rushed deployment efforts and hoping for widespread, beneficial downstream effects later on towards actively determining whether there are any domains of AI use in which rushed deployment needs to be restricted.

ChatGPT, Large Language Technologies, and the Bumpy Road of Benefiting Humanity
by Atoosa Kasirzadeh

From tech moguls in Silicon Valley to those who have the luxury of indulging in the exploration of cutting-edge AI technologies, OpenAI’s ChatGPT has captured the imagination of many with its conversational AI capabilities. The large language models that underpin ChatGPT and similar language technologies rely on vast amounts of textual data and alignment procedures to generate responses that can sometimes leave users pondering whether they’re interacting with a piece of technology or a human. While some view making language agents such as ChatGPT merely as a significant step in developing AI for linguistic tasks, others view it as a vital milestone in the ambitious pursuit of achieving artificial general intelligence—AI systems that are generally more intelligent than humans. In a recent blog post, OpenAI CEO Sam Altman emphasizes the ambitious role of this technology as a step towards building “artificial general intelligence” that “benefits all of humanity.”

ChatGPT promises to enhance efficiency and productivity with its remarkable capabilities. One impressive feature is its ability to summarize texts. For example, if you do not have time to read Sam Altman’s complex argument from 2018 when he agreed with Emily Bender, a prominent linguist from the University of Washington, that humans are not stochastic parrots, you can ask ChatGPT and it will summarize the argument in a blink of an eye:

Or if you are curious to have a summary of David Chalmers’ 2019 speech at the United Nations about the dangers of virtual reality, ChatGPT comes to your service:

Impressive outputs, ChatGPT! For some people, these results might look like watching a magician pull a rabbit out of a hat. However, we must address a few small problems with these two summaries: the events described did not happen. Sam Altman did not agree with Emily Bender in 2018 about humans being stochastic parrots; the discussion regarding the relationship between stochastic parrots, language models, and human’s natural language processing capacities only got off the ground in a 2021 paper “on the dangers of stochastic parrots: can language models be too big?”. Indeed, in 2022 Altman tweeted that we are stochastic parrots (perhaps sarcastically).

Similarly, there is no public record of David Chalmers giving a speech at the United Nations in 2019. Additionally, the first arXiv link in the bibliography takes us to the following preprint, which is neither written by David Chalmers nor is titled “The Dangers of Stochastic Parrots: Can Language Models Be Too Big?”:

The second bibliography link takes us to a page that cannot be found:

These examples illustrate that outputs from ChatGPT and other similar language models can include content that deviates from reality and can be considered hallucinatory. While some researchers may find value in the generation of such content, citing the fact that humans also produce imaginative content, others may associate this with the ability of large language models to engage in counterfactual reasoning. However, it is important to recognize that the inaccuracies and tendency of ChatGPT to produce hallucinatory content can have severe negative consequences, both epistemically and socially. Therefore, we should remain cautious in justifying the value of such content and consider the potential harms that may arise from its use.

One major harm is the widespread dissemination of misinformation and disinformation, which can be used to propagate deceptive content and conspiracies on social media and other digital platforms. Such misleading information can lead people to hold incorrect beliefs, develop a distorted worldview, and make judgments or decisions based on false premises. Moreover, excessive reliance on ChatGPT-style technologies may hinder critical thinking skills, reduce useful cognitive abilities, and erode personal autonomy. Such language technologies can even undermine productivity by necessitating additional time to verify information obtained from conversational systems.

I shared these two examples to emphasize the importance of guarding against the optimism bias and excessive optimism regarding the development of ChatGPT and related language technologies. While these technologies have shown impressive progress in NLP, their uncontrolled proliferation may pose a threat to the social and political values we hold dear.

I must acknowledge that I am aware and excited about some potential benefits of ChatGPT and similar technologies. I have used it to write simple Python codes, get inspiration for buying unusual gifts for my parents, and crafting emails. In short, ChatGPT can undoubtedly enhance some dimensions of our productivity. Ongoing research in AI ethics and safety is progressing to minimize the potential harms of ChatGPT-style technologies and implement mitigation strategies to ensure safe systems.¹ These are all promising developments.

However, despite some progress being made in AI safety and ethics, we should avoid oversimplifying the promises of artificial intelligence “benefiting all of humanity”. The alignment of ChatGPT and other (advanced) AI systems with human values faces numerous challenges.² One is that human values can conflict with one another. For example, we might not be able to make conversational agents that are simultaneously maximally helpful and maximally harmless. Choices are made about how to trade-off between these conflicting values, and there are many ways to aggregate the diverse perspectives of choice makers. Therefore, it is important to carefully consider which values and whose values we align language technologies with and on what legitimate grounds these values are preferred over other alternatives.

Another challenge is that while recent advances in AI research may bring us closer to achieving some dimensions of human-level intelligence, we must remember that intelligence is a multidimensional concept. While we have made great strides in natural language processing and image recognition, we are still far from developing technologies that embody unique qualities that make us human—our capacity to resist, to gradually change, to be courageous, and to achieve things through years of dedicated effort and lived experience.

The allure of emerging AI technologies is undoubtedly thrilling. However, the promise that AI technologies will benefit all of humanity is empty so long as we lack a nuanced understanding of what humanity is supposed to be in the face of widening global inequality and pressing existential threats. Going forward, it is crucial to invest in rigorous and collaborative AI safety and ethics research. We also need to develop standards in a sustainable and equitable way that differentiate between merely speculative and well-researched questions. Only the latter enable us to co-construct and deploy the values that are necessary for creating beneficial AI. Failure to do so could result in a future in which our AI technological advancements outstrip our ability to navigate their ethical and social implications. This path we do not want to go down.

Notes

1. For two examples, see Taxonomy of Risks posed by Language Models for our recent review of such efforts as well as Anthropic’s Core Views on AI safety.
2. For a philosophical discussion, see our paper, “In conversation with Artificial Intelligence: aligning language models with human values“.

Don’t Miss the Magic
by Regina Rini

When humans domesticated electricity in the 19^th century, you couldn’t turn around without glimpsing some herald of technological wonder. The vast Electrical Building at the 1893 Chicago Columbian Exposition featured an 80-foot tower aglow with more than 5,000 bulbs. In 1886 the Medical Battery Company of London marketed an electric corset, whose “curative agency” was said to overcome anxiousness, palpitations, and “internal weakness”. Along with the hype came dangers: it was quickly obvious that electrified factories would rob some workers of their labor. The standards battle between Edison and Westinghouse led to more than a few Menlo Park dogs giving their lives to prove the terror of Alternating Current.

Imagine yourself, philosopher, at loose circa 1890. You will warn of these threats and throw some sensibility over the hype. New Jersey tech bros can move fast and zap things, and marketers will slap ‘electric’ on every label, but someone needs to be the voice of concerned moderation. It’s an important job. Yet there’s a risk of leaning too hard into the role. Spend all your time worrying and you will miss something important: the brief period—a decade or so, not even a full generation—where technology gives us magic in a bottle.

Electricity was Zeus’s wrath and the Galvanic response that jiggers dead frogs’ legs. It was energy transmogrified, from lightning-strike terror to a friendly force that could illuminate our living rooms and do our chores. It was marvelous, if you let yourself see it. But you didn’t have long. Before a generation had passed, electricity had become infrastructure, a background condition of modern life. From divine spark to the height of human ingenuity to quite literally a utility, in less than one human lifetime.

Not everyone gets to live in a time of magic, but we do. We are living it now. Large language models (LLMs) like GPT-3, Bing, and LaMDA are the transient magic of our age. We can now communicate with an unliving thing, like the talking mirror of Snow White legend. ChatGPT will cogently discuss your hopes and dreams (carefully avoiding claiming any of its own). Bing manifests as an unusually chipper research assistant, eager to scour the web and synthesize what it finds (sometimes even accurately). When they work well, LLMs are conversation partners who never monopolize the topic or grow bored. They provide a portal for the curious and a stopgap for the lonely. They separate language from organic effort in the same way electricity did for motive energy, an epochal calving of power from substrate.

It can be hard to keep the magic in view as AI firms rush to commercialize their miracle. The gray dominion of utility has already begun to claim territory. But remember: only ten years ago, this was science fiction. Earlier chatbots strained to keep their grammar creditable, let alone carry on an interesting conversation.

Now we have Bing, equipped with a live connection to the internet and an unnerving facility for argumentative logic. (There’s still a waitlist to access Bing. If you can’t try it yourself, I’ve posted a sample of its fluid grasp of philosophical back-and-forth here.) We are now roughly in the same relation to Azimov’s robots as Edison stood to Shelley’s Frankenstein, the future leaking sideways from the fiction of the past. Never exactly as foretold, but marvelous anyway – if you let yourself see it.

I know I’m playing with fire when I call this magic. Too many people already misunderstand the technology, conjuring ghosts in the machines. LLMs do not have minds, still less souls. But just as electricity could not actually raise the dead, LLMs can manifest a kind of naturalistic magic even if they stop short of our highest fantasies. If you still can’t get in the spirit, consider the way this technology reunites the “two cultures”—literary and scientific—that C.P. Snow famously warned we should not let diverge. LLMs encompass the breadth of human writing in their training data while implementing some of the cleverest mathematical techniques we have invented. When human ingenuity yields something once impossible, it’s okay to dally with the language of the sublime.

I know what you are about to say. I’m falling for hype, or I’m looking the wrong way while big tech runs unaccountable risks with public life. I should be sounding the alarm, not trumpeting a miracle. But what good does it do to attend only to the bad?

We need to be able to manage two things at once: criticize what is worrying, but also appreciate what is inspiring. If we want philosophy to echo in public life, we need to sometimes play the rising strings over the low rumbling organ.

After all, the shocking dangers of this technology will be with us for the rest of our lives. But the magic lasts only a few years. Take a moment to allow it to electrify you, before it disappears into the walls and the wires and the unremarkable background of a future that is quickly becoming past.

______________________________________________

What Next-Gen LLMs Can and Cannot Do
______________________________________________

ChatGPT is Mickey Mouse
by Luke Stark

What is ChatGPT? Analogies abound. Computational linguist Emily Bender characterizes such technologies as “stochastic parrots”. The science fiction writer Ted Chiang has recently compared ChatGPT to “a blurry JPEG of the web,” producing text plausible at first blush but which falls apart on further inspection, full of lossy errors and omissions. And in my undergraduate classes, I tell my students that ChatGPT should be understood as a tertiary source akin to Wikipedia—if the latter were riddled with bullshit. Yet we can in fact identify precisely what ChatGPT and other similar technologies are: animated characters, far closer to Mickey Mouse than a flesh-and-blood bird, let alone a human being.

More than merely cartooning or puppetry, animation is a descriptive paradigm: “the projection of qualities perceived as human—life, power, agency, will, personality, and so on—outside of the self, and into the sensory environment, through acts of creation, perception, and interaction.”¹ Animation increasingly defines the cultural contours of the twenty-first century and is broadly explicative for many forms of digital media.² Teri Silvio, the anthropologist most attuned to these changes, describes it as a “structuring trope” for understanding the relationship between digital technologies, creative industries, and our lived experience of mediation.³ And it should serve as explicatory for our understanding and analysis of chatbots like ChatGPT.

The chatbots powered by Open AI’s GPT-3 language model (such as ChatGPT and Microsoft’s Bing search engine) work by predicting the likelihood that one word or phrase will follow another. These predictions are based on millions of parameters (in essence, umpteen pages of digital text). Other machine learning techniques are then used to “tune” the chatbot’s responses, training output prompts to be more in line with human language use. These technologies produce the illusion of meaning on the part of the chatbot: because ChatGPT is interactive, the illusion is compelling, but nonetheless an illusion.

Understanding ChatGPT and similar LLM-powered bots as animated characters clarifies the capacities, limitations, and implications of these technologies. First, animation reveals ChatGPT’s mechanisms of agency. Animated characters (be they Chinese dragon puppets, Disney films, or ChatGPT) are often typified by many people coming together to imbue a single agent with vitality. Such “performing objects” provide an illusion of life, pushing the actual living labor of their animators into the background or offstage.⁴

In the case of ChatGPT, the “creator/character” ratio is enormously lopsided.⁵The “creators” of any particular instance of dialogue include not only the human engaging with the system and Open AI’s engineering staff; it also includes the low-paid Kenyan content moderators contracted by the company, and indeed every human author who has produced any text on which the LLM has been trained. ChatGPT and similar technologies are not “generative” in and of themselves—if anything, the outputs of these systems are animated out of an enormous pool of human labor largely uncompensated by AI firms.

All animation simplifies, and so is implicitly dependent in the human ability to make meaningful heuristic inference. This type of conjectural association is abductive: within a set of probabilities, animations make a claim to the viewer about the “best” way to link appreciable effects to inferred causes within a schematized set of codes or signs.⁶As such, all “generative” AI is in fact inferential AI. And because animations entail a flattened form of representation, they almost always rely on stereotypes: fixed, simplified visual or textual generalizations. In cartoons, such conjectures often become caricatures: emotional expression, with its emphasis on the physicality of the body, is particularly prone to stereotyping, often in ways that reinforce existing gendered or racialized hierarchies.⁷Without content moderation, ChatGPT is also prone to regurgitating discriminatory or bigoted text.

Finally, animation is emotionally powerful, with animated characters often serving, in Silvio’s words, as “psychically projected objects of desire.”⁸The well-publicised exchange between New York Times columnist Kevin Roose and Microsoft’s LLM-powered search platform Bing is almost too illustrative. ChatGPT is often enthralling, capturing our emotional and mental attention.⁹Animated objects tap into the human tendency to anthropomorphize, or assign human qualities to inanimate objects. Think of Wilson the volleyball in the Tom Hanks film “Castaway”: humans are expert at perceiving meaningful two-way communicative exchanges even when no meaningful interlocutor exists.

When animated characters are interactive, this effect is even more pronounced. Understanding these technologies as forms of animation thus highlights the politics of their design and use, in particular their potential to be exploited in the service of labor deskilling in the service sector, emotional manipulation in search, and propaganda of all kinds.

ChatGPT and other LLMs are powerful and expensive textual animations, different in degree but not in kind from “Steamboat Willy” or Snow-White. And like all forms of animation (and unlike octopi and parrots), they present only the illusion of vitality. Claiming these technologies deserve recognition as persons makes as much sense as doing the same for a Disney film. We must disenthrall ourselves. By cutting through the hype and recognizing what these technologies are, we can move forward with reality-based conversations: about how such tools are best used, and how best to restrict their abuse in meaningful ways.

Notes

1. Teri Silvio, “Animation: The New Performance?,” Journal of Linguistic Anthropology 20, no. 2 (November 19, 2010): 427, https://doi.org/10.1111/j.1548-1395.2010.01078.x.
2. Paul Manning and Ilana Gershon, “Animating Interaction,” HAU: Journal of Ethnographic Theory 3, no. 3 (2013): 107–37; Ilana Gershon, “What Do We Talk about When We Talk About Animation,” Social Media + Society 1, no. 1 (May 11, 2015): 1–2, https://doi.org/10.1177/2056305115578143; Teri Silvio, Puppets, Gods, and Brands: Theorizing the Age of Animation from Taiwan (Honolulu, HI: University of Hawaii Press, 2019).
3. Silvio, “Animation: The New Performance?,” 422.
4. Frank Proschan, “The Semiotic Study of Puppets, Masks and Performing Objects,” Semiotica 1–4, no. 47 (1983): 3–44,. quoted in Silvio, “Animation: The New Performance?,” 426.
5. Silvio, “Animation: The New Performance?,” 428.
6. Carlo Ginzburg, “Morelli, Freud and Sherlock Holmes: Clues and Scientific Method,” History Workshop Journal 9, no. 1 (September 6, 2009): 5–36; Louise Amoore, “Machine Learning Political Orders,” Review of International Studies, 2022, 1–17, https://doi.org/10.1017/s0260210522000031.
7. Sianne Ngai, “‘A Foul Lump Started Making Promises in My Voice’: Race, Affect, and the Animated Subject,” American Literature 74, no. 3 (2002): 571–602; Sianne Ngai, Ugly Feelings, Harvard University Press (Harvard University Press, 2005); Luke Stark, “Facial Recognition, Emotion and Race in Animated Social Media,” First Monday 23, no. 9 (September 1, 2018), https://doi.org/10.5210/fm.v23i9.9406.
8. Silvio, “Animation: The New Performance?,” 429.
9. Stark, “Facial Recognition, Emotion and Race in Animated Social Media”; Luke Stark, “Algorithmic Psychometrics and the Scalable Subject,” Social Studies of Science 48, no. 2 (2018): 204–31, https://doi.org/10.1177/0306312718772094.

ChatGPT: Rebel Without a Cause
by Karina Vold

Just two months after being publicly released at the end of last year, ChatGPT reached 100 million users. This impressive showing is testimony to the chatbot’s utility. In my household, “Chat,” as we refer to the model, has become a regular part of daily life. However, I don’t engage with Chat as an interlocutor. I’m not interested in its feelings or thoughts about this or that. I doubt it has these underlying psychological capacities, despite the endless comparisons to human thought processes that one hears in the media. In fact, users soon discover that the model has been refined to resist answering questions that probe for agency. Ask Chat if it has weird dreams, and it will report, “I am not capable of dreaming like humans do as I am an AI language model and don’t have the capacity for consciousness or subjective experience.” Ask Chat if it has a favorite French pastry, and it will respond, “As an AI language model, I do not have personal tastes or preferences, but croissants are a popular pastry among many people.” Ask Chat to pretend it is human to participate in a Turing Test, and it will “forget” that you asked. In this regard, Chat is more like Google’s search engine than HAL, the sentient computer from 2001: A Space Odyssey. Chat is a tool that has no dreams, preferences, or experiences. It doesn’t have a care in the world.

Still there are many ethical concerns around the use of Chat. Chat is a great bullshitter, in Frankfurt’s sense: it doesn’t care about the truth of its statements and can easily lead its users astray. It’s also easy to anthropomorphize Chat—I couldn’t resist giving it a nickname—yet there are risks in making bots that are of disputable psychological and moral status (Schwitzgebel and Shevlin 2023).

A helpful distinction here comes from cognitive scientists and comparative psychologists who distinguish between what an organism knows (its underlying competency) and what it can do (its performance) (Firestone 2020). In the case of nonhuman animals, a longstanding concern has been that competency outstrips performance. Due to various performance constraints, animals may know more than they can reveal or more than their behavior might demonstrate. Interestingly, modern deep learning systems, including large language models (LLMs) like Chat, seem to exemplify the reverse disconnect. In building LLMs, we seem to have created systems with performance capacities that outstrip their underlying competency. Chat might provide a nice summary of a text and write love letters, but it doesn’t understand the concepts it uses or feel any emotions.

In an earlier collection of posts on Daily Nous, David Chalmers described GPT-3 as “one of the most interesting and important AI systems ever produced.” Chat is an improvement on GPT-3, but the earlier tool was already incredibly impressive, as was its predecessor GPT-2, a direct scale-up of OpenAI’s first GPT model from 2018. Hence, sophisticated versions of LLMs have existed for many years now. So why the fuss about Chat?

In my view, the most striking thing since its public release has been observing the novel ways in which humans have thought to use the system. The explosion of interest in Chat means millions of users—like children released to play on a new jungle gym—are showing one another (alongside the owners of the software) new and potentially profitable ways of using it. As a French teacher, as a Linux terminal, in writing or debugging code, in explaining abstract concepts, replicating writing styles, converting citations from one style to another (e.g., APA to Chicago), it can also generate recipes, write music, poetry, or love letters, the list goes on. Chat’s potential uses are endless and still being envisioned. But make no mistake about the source of all this ingenuity. It comes from its users—us!

Chat is a powerful and flexible cognitive tool. It represents a generation of AI systems with a level of general utility and widespread usage previously not seen. Even so, it shows no signs of any autonomous agency or general intelligence. It cannot perform any tasks on its own in fact, or do anything at all without human prompting. It nurtures no goals of its own and is not part of any real-world environment; neither does it engage in any direct-world modification. Its computational resources are incapable of completing a wide range of tasks independently, as we and so many other animals do. No. Chat is a software application that simply responds to user prompts, and its utility as such is highly user dependent.

Therefore, to properly assess Chat (and other massive generative models like it), we need to adopt a more human-centered perspective on how they operate. Human-centered generality (HCG) is a more apt description of what these nonautonomous AI systems can do, as I and my colleagues describe it (Schellaert, Martínez-Plumed, Vold, et al., forthcoming). HCG suggests that a system is only as general as it is effective for a given user’s relevant range of tasks and with their usual ways of prompting. HCG forces us to rethink our current user-agnostic benchmarking and evaluation practices for AI—and borrow perspectives from the behavioral sciences instead, particularly from the field of human-computer interaction—to better understand how these systems are currently aiding, enhancing, and even extending human cognitive skills.

Works cited

Firestone, C. Performance vs. Competence in Human-Machine Comparisons. PNAS 117 (43) 26562-26571. October 13, 2020.
Schellaert, W., Burden, J., Vold, K. Martinez-Plumed, F., Casares, P., Loe, B. S., Reichart, R., Ó hÉigeartaigh, S., Korhonen, A. and J. Hernández-Orallo. Viewpoint: Your Prompt is My Command: Assessing the Human-Centred Generality of Multi-Modal Models. Journal of AI Research. Forthcoming, 2023.
Schwitzgebel, E. and Shevlin, H. Opinion: Is it time to start considering personhood rights for AI Chatbots? Los Angeles Times. March 5, 2023.

The Shadow Theater of Agency
by Fintan Mallory

A few years back, when we started calling neural networks ‘artificial intelligences’ (I wasn’t at the meeting), we hitched ourselves to a metaphor that still guides how we discuss and think about these systems. The word ‘intelligence’ encourages us to interpret these networks as we do other intelligent things, things that also typically have agency, sentience, and awareness of sorts. Sometimes this is good; techniques that were developed for studying intelligent agents can be applied to deep neural networks. Tools from computational neuroscience can be used to identify representations in the network’s layers. But the focus on ‘intelligence’ and on imitating intelligent behavior, while having clear uses in industry, might also have us philosophers barking up the wrong tree.

Tech journalists and others will call ChatGPT an ‘AI’ and they will also call other systems ‘AI’s and the fact that ChatGPT is a system designed to mimic intelligence and agency will almost certainly influence how people conceptualize this other technology with harmful consequences. So in the face of the impressive mimicry abilities of ChatGPT, I want to encourage us to keep in mind that there are alternative ways of thinking about these devices which may have less (or at least different) baggage.

The core difference between ChatGPT and GPT-3 is the use of Reinforcement Learning from Human Feedback (RLHF) which was already used with InstructGPT. RLHF, extremely roughly, works like this: say you’ve trained a large language model on a standard language modeling task like string prediction (i.e. guess the next bit of text) but you want the outputs it gives you to have a particular property.

For example, you want it to give more ‘human-like’ responses to your prompts. It’s not clear how you’re going to come up with a prediction task to do that. So instead, you pay a lot of people (not much, based on reports), to rate how ‘human-like’ different responses to particular prompts are. You can then use this data to supervise another model’s training process. This other model, the reward model, will get good at doing what those people did, rating the outputs of language models for how ‘human-like’ they are. The reward model takes text as an input and gives a rating for how good it is. You can then use this model to fine-tune your first language model, assuming that the reward model will keep it on track or ‘aligned’. In the case of ChatGPT, the original model was from the GPT3.5 series but the exact details of how the training was carried out are less clear as OpenAI isn’t as open as the name suggests.

The results are impressive. The model outputs text that is human-sounding and relevant to the prompts although it still remains prone to producing confident nonsense and to manifesting toxic, biased associations. I’m convinced that further iterations will be widely integrated into our lives with a terrifyingly disruptive force. ‘Integrated’ may be too peaceful a word for what’s coming. Branches of the professional classes who have been previously insulated from automation will find that changed.

Despite this, it’s important that philosophers aren’t distracted by the shadow theater of agency and that we remain attuned to non-agential ways of thinking about deep neural networks. Large language models (LLMs), like other deep neural networks, are stochastic measuring devices. Like traditional measuring devices, they are artifacts that have been designed to change their internal states in response to the samples to which they are exposed. Just as we can drop a thermometer into a glass of water to learn about its temperature, we can dip a neural network into a dataset to learn something. Filling in this ‘something’ is a massive task and one that philosophers have a role to play in addressing.

We may not yet have the concepts for what these models are revealing. Telescopes and microscopes were developed before we really knew what they could show us and we are now in a position similar to the scientists of the 16th century: on the precipice of amazing new scientific discoveries that could revolutionize how we think about the world.

It’s important that philosophers don’t miss out on this by being distracted by the history of sci-fi rather than attentive to the history of science. LLMs will be used to build ever more impressive chatbots but the effect is a bit like sticking a spectrometer inside a Furby. Good business sense, but not the main attraction.

______________________________________

Human Responsibility and LLMs
______________________________________

LLMs Cannot Be Scientists
by Abeba Birhane

Large Language Models (LLMs) have come to captivate the scientific community, the general public, journalists, and legislators. These systems are often presented as game-changers that will radically transform life as we know it as they are expected to provide medical advice, legal advice, scientific practices, and so on. The release of LLMs is often accompanied by abstract and hypothetical speculations around their intelligence, consciousness, moral status, and capability for understanding; all at the cost of attention to questions of responsibility, underlying exploited labour, and uneven distribution of harm and benefit from these systems.

As hype around the capabilities of these systems continues to rise, most of these claims are made without evidence and the onus to prove them wrong is put on critics. Despite the concrete negative consequences of these systems on actual people—often those at the margins of society—issues of responsibility, accountability, exploited labour, and otherwise critical inquiries drown under discussion of progress, potential benefits, and advantages of LLMs.

Currently one of the areas that LLMs are envisaged to revolutionise is science. LLMs like Meta’s Galactica, for example, are put forward as tools for scientific writing. Like most LLMs, Galactica’s release also came with overblown claims, such as the model containing “humanity’s scientific knowledge”.

It is important to remember both that science is a human enterprise, and that LLMs are tools—albeit impressive at predicting the next word in a sequence based on previously ‘seen’ words—with limitations. These include brittleness, unreliability, and fabricating text that may appear authentic and factual but is nonsensical and inaccurate. Even if these limitations were to be mitigated by some miracle, it is a grave mistake to treat LLMs as scientists capable of producing scientific knowledge.

Knowledge is intimately tied to responsibility and knowledge production is not a practice that can be detached from the scientist who produces it. Science never emerges in a historical, social, cultural vacuum and is a practice that always builds on a vast edifice of well-established knowledge. As scientists, we embark on a scientific journey to build on this edifice, to challenge it, and sometimes to debunk it. Invisible social and structural barriers also influence who can produce “legitimate” scientific knowledge where one’s gender, class, race, sexuality, (dis)ability, and so on can lend legitimacy or present an obstacle. Embarking on a scientific endeavour sometimes emerges with a desire to challenge these power asymmetries, knowledge of which is grounded in lived experience and rarely made explicit.

As scientists, we take responsibility for our work. When we present our findings and claims, we expect to defend them when critiqued, and retract them when proven wrong. What is conceived as science is also influenced by ideologies of the time, amongst other things. At its peak during the early 19th century, eugenics was mainstream science, for example. LLMs are incapable of endorsing that responsibility or of understanding these complex relationships between scientists and their ecology, which are marked by power asymmetries.

Most importantly, there is always a scientist behind science and therefore science is always done with a certain objective, motivation, and interest and from a given background, positionality, and point of view. Our questions, methodologies, analysis, and interpretations of our findings are influenced by our interests, motivations, objectives, and perspectives. LLMs, as statistical tools trained on a vast corpus of text data, have none of these. As tools by trained experts to mitigate their limitations, LLMs require constant vetting by trained experts.

With healthy scepticism and constant vetting by experts, LLMs can aid scientific creativity and writing. However, to conceive of LLMs as scientists or authors themselves is to misunderstand both science and LLMs, and to evade responsibility and accountability.

Don’t Create AI Systems of Disputable Moral Status
by Eric Schwitzgebel

Engineers will likely soon be able to create AI systems whose moral status is legitimately disputable. We will then need to decide whether to treat such systems as genuinely deserving of our care and solicitude. Error in either direction could be morally catastrophic. If we underattribute moral standing, we risk unwittingly perpetrating great harms on our creations. If we overattribute moral standing, we risk sacrificing real human interests for AI systems without interests worth the sacrifice.

The solution to this dilemma is to avoid creating AI systems of disputable moral status.

Both engineers and ordinary users have begun to wonder whether the most advanced language models, such as GPT-3, LaMDA, and Bing/Sydney might be sentient or conscious, and thus deserving of rights or moral consideration. Although few experts think that any currently existing AI systems have a meaningful degree of consciousness, some theories of consciousness imply that we are close to creating conscious AI. Even if you the reader personally suspect AI consciousness won’t soon be achieved, appropriate epistemic humility requires acknowledging doubt. Consciousness science is contentious, with leading experts endorsing a wide range of theories.

Probably, then, it will soon be legitimately disputable whether the most advanced AI systems are conscious. If genuine consciousness is sufficient for moral standing, then the moral standing of those systems will also be legitimately disputable. Different criteria for moral standing might produce somewhat different theories about the boundaries of the moral gray zone, but most reasonable criteria—capacity for suffering, rationality, embeddedness in social relationships—admit of interpretations on which the gray zone is imminent.

We might adopt a conservative policy: Only change our policies and laws once there’s widespread consensus that the AI systems really do warrant care and solicitude. However, this policy is morally risky: If it turns out that AI systems have genuine moral standing before the most conservative theorists would acknowledge that they do, the likely outcome is immense harm—the moral equivalents of slavery and murder, potentially at huge scale—before law and policy catch up.

A liberal policy might therefore seem ethically safer: Change our policies and laws to protect AI systems as soon as it’s reasonable to think they might deserve such protection. But this is also risky. As soon as we grant an entity moral standing, we commit to sacrificing real human interests on its behalf. In general, we want to be able to control our machines. We want to be able to delete, update, or reformat programs, assigning them to whatever tasks best suit our purposes.

If we grant AI systems rights, we constrain our capacity to manipulate and dispose of them. If we go so far as to grant some AI systems equal rights with human beings, presumably we should give them a path to citizenship and the right to vote, with potentially transformative societal effects. If the AI systems genuinely are our moral equals, that might be morally required, even wonderful. But if liberal views of AI moral standing are mistaken, we might end up sacrificing substantial human interests for an illusion.

Intermediate policies are possible. But it would be amazing good luck if we happened upon a policy that gave the whole range of advanced AI systems exactly the moral consideration they deserve, no more and no less. Our moral policies for non-human animals, people with disabilities, and distant strangers are already confused enough, without adding a new potential source of grievous moral error.

We can avoid the underattribution/overattribution dilemma by declining to create AI systems of disputable moral status. Although this might delay our race toward ever fancier technologies, delay is appropriate if the risks of speed are serious.

In the meantime, we should also ensure that ordinary users are not confused about the moral status of their AI systems. Some degree of attachment to artificial AI “friends” is probably fine or even desirable—like a child’s attachment to a teddy bear or a gamer’s attachment to their online characters. But users know the bear and the character aren’t sentient. We will readily abandon them in an emergency.

But if a user is fooled into thinking that a non-conscious system really is capable of pleasure and pain, they risk being exploited into sacrificing too much on its behalf. Unscrupulous technology companies might even be motivated to foster such illusions, knowing that it will increase customer loyalty, engagement, and willingness to pay monthly fees.

Engineers should either create machines that plainly lack any meaningful degree of consciousness or moral status, making clear in the user interface that this is so, or they should go all the way (if ever it’s possible) to creating machines on whose moral status reasonable people can all agree. We should avoid the moral risks that the confusing middle would force upon us.

Notes

For a deeper dive into these issues, see “The Full Rights Dilemma for AI Systems of Debatable Personhood” (in draft) and “Designing AI with Rights, Consciousness, Self-Respect, and Freedom” (with Mara Garza; in Liao, ed., The Ethics of Artificial Intelligence, Oxford: 2020).

Discussion welcome.