Login
“There are good reasons to think that some AIs today have wellbeing.”
In this guest post, Simon Goldstein (Dianoia Institute, Australian Catholic University) and Cameron Domenico Kirk-Giannini (Rutgers University – Newark, Center for AI Safety) argue that some existing artificial intelligences have a kind of moral significance because they’re beings for whom things can go well or badly.
This is the sixth in a series of weekly guest posts by different authors at Daily Nous this summer.
[Posts in the summer guest series will remain pinned to the top of the page for the week in which they’re published.]
We recognize one another as beings for whom things can go well or badly, beings whose lives may be better or worse according to the balance they strike between goods and ills, pleasures and pains, desires satisfied and frustrated. In our more broad-minded moments, we are willing to extend the concept of wellbeing also to nonhuman animals, treating them as independent bearers of value whose interests we must consider in moral deliberation. But most people, and perhaps even most philosophers, would reject the idea that fully artificial systems, designed by human engineers and realized on computer hardware, may similarly demand our moral consideration. Even many who accept the possibility that humanoid androids in the distant future will have wellbeing would resist the idea that the same could be true of today’s AI.
Perhaps because the creation of artificial systems with wellbeing is assumed to be so far off, little philosophical attention has been devoted to the question of what such systems would have to be like. In this post, we suggest a surprising answer to this question: when one integrates leading theories of mental states like belief, desire, and pleasure with leading theories of wellbeing, one is confronted with the possibility that the technology already exists to create AI systems with wellbeing. We argue that a new type of AI—the artificial language agent—has wellbeing. Artificial language agents augment large language models with the capacity to observe, remember, and form plans. We also argue that the possession of wellbeing by language agents does not depend on them being phenomenally conscious. Far from a topic for speculative fiction or future generations of philosophers, then, AI wellbeing is a pressing issue. This post is a condensed version of our argument. To read the full version, click here.
Artificial language agents (or simply language agents) are our focus because they support the strongest case for wellbeing among existing AIs. Language agents are built by wrapping a large language model (LLM) in an architecture that supports long-term planning. An LLM is an artificial neural network designed to generate coherent text responses to text inputs (ChatGPT is the most famous example). The LLM at the center of a language agent is its cerebral cortex: it performs most of the agent’s cognitive processing tasks. In addition to the LLM, however, a language agent has files that record its beliefs, desires, plans, and observations as sentences of natural language. The language agent uses the LLM to form a plan of action based on its beliefs and desires. In this way, the cognitive architecture of language agents is familiar from folk psychology.
For concreteness, consider the language agents built this year by a team of researchers at Stanford and Google. Like video game characters, these agents live in a simulated world called ‘Smallville’, which they can observe and interact with via natural-language descriptions of what they see and how they act. Each agent is given a text backstory that defines their occupation, relationships, and goals. As they navigate the world of Smallville, their experiences are added to a “memory stream” in the form of natural language statements. Because each agent’s memory stream is long, agents use their LLM to assign importance scores to their memories and to determine which memories are relevant to their situation. Then the agents reflect: they query the LLM to make important generalizations about their values, relationships, and other higher-level representations. Finally, they plan: They feed important memories from each day into the LLM, which generates a plan for the next day. Plans determine how an agent acts, but can be revised on the fly on the basis of events that occur during the day. In this way, language agents engage in practical reasoning, deciding how to promote their goals given their beliefs.
The conclusion that language agents have beliefs and desires follows from many of the most popular theories of belief and desire, including versions of dispositionalism, interpretationism, and representationalism.
According to the dispositionalist, to believe or desire that something is the case is to possess a suitable suite of dispositions. According to ‘narrow’ dispositionalism, the relevant dispositions are behavioral and cognitive; ‘wide’ dispositionalism also includes dispositions to have phenomenal experiences. While wide dispositionalism is coherent, we set it aside here because it has been defended less frequently than narrow dispositionalism.
Consider belief. In the case of language agents, the best candidate for the state of believing a proposition is the state of having a sentence expressing that proposition written in the memory stream. This state is accompanied by the right kinds of verbal and nonverbal behavioral dispositions to count as a belief, and, given the functional architecture of the system, also the right kinds of cognitive dispositions. Similar remarks apply to desire.
According to the interpretationist, what it is to have beliefs and desires is for one’s behavior (verbal and nonverbal) to be interpretable as rational given those beliefs and desires. There is no in-principle problem with applying the methods of radical interpretation to the linguistic and nonlinguistic behavior of a language agent to determine what it believes and desires.
According to the representationalist, to believe or desire something is to have a mental representation with the appropriate causal powers and content. Representationalism deserves special emphasis because “probably the majority of contemporary philosophers of mind adhere to some form of representationalism about belief” (Schwitzgebel).
It is hard to resist the conclusion that language agents have beliefs and desires in the representationalist sense. The Stanford language agents, for example, have memories which consist of text files containing natural language sentences specifying what they have observed and what they want. Natural language sentences clearly have content, and the fact that a given sentence is in a given agent’s memory plays a direct causal role in shaping its behavior.
Many representationalists have argued that human cognition should be explained by positing a “language of thought.” Language agents also have a language of thought: their language of thought is English!
An example may help to show the force of our arguments. One of Stanford’s language agents had an initial description that included the goal of planning a Valentine’s Day party. This goal was entered into the agent’s planning module. The result was a complex pattern of behavior. The agent met with every resident of Smallville, inviting them to the party and asking them what kinds of activities they would like to include. The feedback was incorporated into the party planning.
To us, this kind of complex behavior clearly manifests a disposition to act in ways that would tend to bring about a successful Valentine’s Day party given the agent’s observations about the world around it. Moreover, the agent is ripe for interpretationist analysis. Their behavior would be very difficult to explain without referencing the goal of organizing a Valentine’s Day party. And, of course, the agent’s initial description contained a sentence with the content that its goal was to plan a Valentine’s Day party. So, whether one is attracted to narrow dispositionalism, interpretationism, or representationalism, we believe the kind of complex behavior exhibited by language agents is best explained by crediting them with beliefs and desires.
What makes someone’s life go better or worse for them? There are three main theories of wellbeing: hedonism, desire satisfactionism, and objective list theories. According to hedonism, an individual’s wellbeing is determined by the balance of pleasure and pain in their life. According to desire satisfactionism, an individual’s wellbeing is determined by the extent to which their desires are satisfied. According to objective list theories, an individual’s wellbeing is determined by their possession of objectively valuable things, including knowledge, reasoning, and achievements.
On hedonism, to determine whether language agents have wellbeing, we must determine whether they feel pleasure and pain. This in turn depends on the nature of pleasure and pain.
There are two main theories of pleasure and pain. According to phenomenal theories, pleasures are phenomenal states. For example, one phenomenal theory of pleasure is the distinctive feeling theory. The distinctive feeling theory says that there is a particular phenomenal experience of pleasure that is common to all pleasant activities. We see little reason why language agents would have representations with this kind of structure. So if this theory of pleasure were correct, then hedonism would predict that language agents do not have wellbeing.
The main alternative to phenomenal theories of pleasure is attitudinal theories. In fact, most philosophers of wellbeing favor attitudinal over phenomenal theories of pleasure (Bramble). One attitudinal theory is the desire-based theory: experiences are pleasant when they are desired. This kind of theory is motivated by the heterogeneity of pleasure: a wide range of disparate experiences are pleasant, including the warm relaxation of soaking in a hot tub, the taste of chocolate cake, and the challenge of completing a crossword. While differing in intrinsic character, all of these experiences are pleasant when desired.
If pleasures are desired experiences and AIs can have desires, it follows that AIs can have pleasure if they can have experiences. In this context, we are attracted to a proposal defended by Schroeder: an agent has a pleasurable experience when they perceive the world being a certain way, and they desire the world to be that way. Even if language agents don’t presently have such representations, it would be possible to modify their architecture to incorporate them. So some versions of hedonism are compatible with the idea that language agents could have wellbeing.
We turn now from hedonism to desire satisfaction theories. According to desire satisfaction theories, your life goes well to the extent that your desires are satisfied. We’ve already argued that language agents have desires. If that argument is right, then desire satisfaction theories seem to imply that language agents can have wellbeing.
According to objective list theories of wellbeing, a person’s life is good for them to the extent that it instantiates objective goods. Common components of objective list theories include friendship, art, reasoning, knowledge, and achievements. For reasons of space, we won’t address these theories in detail here. But the general moral is that once you admit that language agents possess beliefs and desires, it is hard not to grant them access to a wide range of activities that make for an objectively good life. Achievements, knowledge, artistic practices, and friendship are all caught up in the process of making plans on the basis of beliefs and desires.
Generalizing, if language agents have beliefs and desires, then most leading theories of wellbeing suggest that their desires matter morally.
We’ve argued that language agents have wellbeing. But there is a simple challenge to this proposal. First, language agents may not be phenomenally conscious — there may be nothing it feels like to be a language agent. Second, some philosophers accept:
The Consciousness Requirement. Phenomenal consciousness is necessary for having wellbeing.
The Consciousness Requirement might be motivated in either of two ways: First, it might be held that every welfare good itself requires phenomenal consciousness (this view is known as experientialism). Second, it might be held that though some welfare goods can be possessed by beings that lack phenomenal consciousness, such beings are nevertheless precluded from having wellbeing because phenomenal consciousness is necessary to have wellbeing.
We are not convinced. First, we consider it a live question whether language agents are or are not phenomenally conscious (see Chalmers for recent discussion). Much depends on what phenomenal consciousness is. Some theories of consciousness appeal to higher-order representations: you are conscious if you have appropriately structured mental states that represent other mental states. Sufficiently sophisticated language agents, and potentially many other artificial systems, will satisfy this condition. Other theories of consciousness appeal to a ‘global workspace’: an agent’s mental state is conscious when it is broadcast to a range of that agent’s cognitive systems. According to this theory, language agents will be conscious once their architecture includes representations that are broadcast widely. The memory stream of Stanford’s language agents may already satisfy this condition. If language agents are conscious, then the Consciousness Requirement does not pose a problem for our claim that they have wellbeing.
Second, we are not convinced of the Consciousness Requirement itself. We deny that consciousness is required for possessing every welfare good, and we deny that consciousness is required in order to have wellbeing.
With respect to the first issue, we build on a recent argument by Bradford, who notes that experientialism about welfare is rejected by the majority of philosophers of welfare. Cases of deception and hallucination suggest that your life can be very bad even when your experiences are very good. This has motivated desire satisfaction and objective list theories of wellbeing, which often allow that some welfare goods can be possessed independently of one’s experience. For example, desires can be satisfied, beliefs can be knowledge, and achievements can be achieved, all independently of experience.
Rejecting experientialism puts pressure on the Consciousness Requirement. If wellbeing can increase or decrease without conscious experience, why would consciousness be required for having wellbeing? After all, it seems natural to hold that the theory of wellbeing and the theory of welfare goods should fit together in a straightforward way:
Simple Connection. An individual can have wellbeing just in case it is capable of possessing one or more welfare goods.
Rejecting experientialism but maintaining Simple Connection yields a view incompatible with the Consciousness Requirement: the falsity of experientialism entails that some welfare goods can be possessed by non-conscious beings, and Simple Connection guarantees that such non-conscious beings will have wellbeing.
Advocates of the Consciousness Requirement who are not experientialists must reject Simple Connection and hold that consciousness is required to have wellbeing even if it is not required to possess particular welfare goods. We offer two arguments against this view.
First, leading theories of the nature of consciousness are implausible candidates for necessary conditions on wellbeing. For example, it is implausible that higher-order representations are required for wellbeing. Imagine an agent who has first order beliefs and desires, but does not have higher order representations. Why should this kind of agent not have wellbeing? Suppose that desire satisfaction contributes to wellbeing. Granted, since they don’t represent their beliefs and desires, they won’t themselves have opinions about whether their desires are satisfied. But the desires still are satisfied. Or consider global workspace theories of consciousness. Why should an agent’s degree of cognitive integration be relevant to whether their life can go better or worse?
Second, we think we can construct chains of cases where adding the relevant bit of consciousness would make no difference to wellbeing. Imagine an agent with the body and dispositional profile of an ordinary human being, but who is a ‘phenomenal zombie’ without any phenomenal experiences. Whether or not its desires are satisfied or its life instantiates various objective goods, defenders of the Consciousness Requirement must deny that this agent has wellbeing. But now imagine that this agent has a single persistent phenomenal experience of a homogenous white visual field. Adding consciousness to the phenomenal zombie has no intuitive effect on wellbeing: if its satisfied desires, achievements, and so forth did not contribute to its wellbeing before, the homogenous white field should make no difference. Nor is it enough for the consciousness to itself be something valuable: imagine that the phenomenal zombie always has a persistent phenomenal experience of mild pleasure. To our judgment, this should equally have no effect on whether the agent’s satisfied desires or possession of objective goods contribute to its wellbeing. Sprinkling pleasure on top of the functional profile of a human does not make the crucial difference. These observations suggest that whatever consciousness adds to wellbeing must be connected to individual welfare goods, rather than some extra condition required for wellbeing: rejecting Simple Connection is not well motivated. Thus the friend of the Consciousness Requirement cannot easily avoid the problems with experientialism by falling back on the idea that consciousness is a necessary condition for having wellbeing.
We’ve argued that there are good reasons to think that some AIs today have wellbeing. But our arguments are not conclusive. Still, we think that in the face of these arguments, it is reasonable to assign significant probability to the thesis that some AIs have wellbeing.
In the face of this moral uncertainty, how should we act? We propose extreme caution. Wellbeing is one of the core concepts of ethical theory. If AIs can have wellbeing, then they can be harmed, and this harm matters morally. Even if the probability that AIs have wellbeing is relatively low, we must think carefully before lowering the wellbeing of an AI without producing an offsetting benefit.
[Image made with DALL-E]
Some related posts:
“Philosophers on GPT-3”
“Philosophers on Next-Generation Large Language Models”
“GPT-4 and the Question of Intelligence”
“We’re Not Ready for the AI on the Horizon, But People Are Trying”
“Researchers Call for More Work on Consciousness”
“Dennett on AI: We Must Protect Ourselves Against ‘Counterfeit People’”
“Philosophy, AI, and Society Listserv”
“Talking Philosophy with Chat-GPT“
The post A Case for AI Wellbeing (guest post) first appeared on Daily Nous.
“Lying” in computer-generated texts: hallucinations and omissions
There is huge excitement about ChatGPT and other large generative language models that produce fluent and human-like texts in English and other human languages. But these models have one big drawback, which is that their texts can be factually incorrect (hallucination) and also leave out key information (omission).
In our chapter for The Oxford Handbook of Lying, we look at hallucinations, omissions, and other aspects of “lying” in computer-generated texts. We conclude that these problems are probably inevitable.
Omissions are inevitable because a computer system cannot cram all possibly-relevant information into a text that is short enough to be actually read. In the context of summarising medical information for doctors, for example, the computer system has access to a huge amount of patient data, but it does not know (and arguably cannot know) what will be most relevant to doctors.
Hallucinations are inevitable because of flaws in computer systems, regardless of the type of system. Systems which are explicitly programmed will suffer from software bugs (like all software systems). Systems which are trained on data, such as ChatGPT and other systems in the Deep Learning tradition, “hallucinate” even more. This happens for a variety of reasons. Perhaps most obviously, these systems suffer from flawed data (e.g., any system which learns from the Internet will be exposed to a lot of false information about vaccines, conspiracy theories, etc.). And even if a data-oriented system could be trained solely on bona fide texts that contain no falsehoods, its reliance on probabilistic methods will mean that word combinations that are very common on the Internet may also be produced in situations where they result in false information.
Suppose, for example, on the Internet, the word “coughing” is often followed by “… and sneezing.” Then a patient may be described falsely, by a data-oriented system, as “coughing and sneezing” in situations where they cough without sneezing. Problems of this kind are an important focus for researchers working on generative language models. Where this research will lead us is still uncertain; the best one can say is that we can try to reduce the impact of these issues, but we have no idea how to completely eliminate them.
“Large generative language models’ texts can be factually incorrect (hallucination) and leave out key information (omission).”
The above focuses on unintentional-but-unavoidable problems. There are also cases where a computer system arguably should hallucinate or omit information. An obvious example is generating marketing material, where omitting negative information about a product is expected. A more subtle example, which we have seen in our own work, is when information is potentially harmful and it is in users’ best interests to hide or distort it. For example, if a computer system is summarising information about sick babies for friends and family members, it probably should not tell an elderly grandmother with a heart condition that the baby may die, since this could trigger a heart attack.
Now that the factual accuracy of computer-generated text draws so much attention from society as a whole, the research community is starting to realize more clearly than before that we only have a limited understanding of what it means to speak the truth. In particular, we do not know how to measure the extent of (un)truthfulness in a given text.
To see what we mean, suppose two different language models answer a user’s question in two different ways, by generating two different answer texts. To compare these systems’ performance, we would need a “score card” that allowed us to objectively score the two texts as regards their factual correctness, using a variety of rubrics. Such a score card would allow us to record how often each type of error occurs in a given text, and aggregate the result into an overall truthfulness score for that text. Of particular importance would be the weighing of errors: large errors (e.g., a temperature reading that is very far from the actual temperature) should weigh more heavily than small ones, key facts should weigh more heavily than side issues, and errors that are genuinely misleading should weigh more heavily than typos that readers can correct by themselves. Essentially, the score card would work like a fair school teacher who marks pupils’ papers.
We have developed protocols for human evaluators to find factual errors in generated texts, as have other researchers, but we cannot yet create a score card as described above because we cannot assess the impact of individual errors.
What is needed, we believe, is a new strand of linguistically informed research, to tease out all the different parameters of “lying” in a manner that can inform the above-mentioned score cards, and that may one day be implemented into a reliable fact-checking protocol or algorithm. Until that time, those of us who are trying to assess the truthfulness of ChatGPT will be groping in the dark.
Featured image by Google DeepMind Via Unsplash (public domain)
Real patterns and the structure of language
There’s been a lot of hype recently about the emergence of technologies like ChatGPT and the effects they will have on science and society. Linguists have been especially curious about what highly successful large language models (LLMs) mean for their business. Are these models unearthing the hidden structure of language itself or just marking associations for predictive purposes?
In order to answer these sorts of questions we need to delve into the philosophy of what language is. For instance, if Language (with a big “L”) is an emergent human phenomenon arising from our communicative endeavours, i.e. a social entity, then AI is still some ways off approaching it in a meaningful way. If Chomsky, and those who follow his work, are correct that language is a modular mental system innately given to human infants and activated by miniscule amounts of external stimulus, then AI is again unlikely to be linguistic, since most of our most impressive LLMs are sucking up so many resources (both in terms of data and energy) that they are far from this childish learning target. On the third hand, if languages are just very large (possibly infinite) collections of sentences produced by applying discrete rules, then AI could be super-linguistic.
In my new book, I attempt to find a middle ground or intersection between these views. I start with an ontological picture (meaning a picture of what there is “out there”) advocated in the early nineties by the prominent philosopher and cognitive scientist, Daniel Dennett. He draws from information theory to distinguish between noise and patterns. In the noise, nothing is predictable, he says. But more often than not, we can and do find regularities in large data structures. These regularities provide us with the first steps towards pattern recognition. Another way to put this is that if you want to send a message and you need the entire series (string or bitmap) of information to do so, then it’s random. But if there’s some way to compress the information, it’s a pattern! What makes a pattern real, is whether or not it needs an observer for its existence. Dennett uses this view to make a case for “mild realism” about the mind and the position (which he calls the “intentional stance”) we use to identify minds in other humans, non-humans, and even artifacts. Basically, it’s like a theory we use to predict behaviour based on the success of our “minded” vocabulary comprising beliefs, desires, thoughts, etc. For Dennett, prediction matters theoretically!
If it’s not super clear yet, consider a barcode. At first blush, the black lines of varying length set to a background of white might seem random. But the lines (and spaces) can be set at regular intervals to reveal an underlying pattern that can be used to encode information (about the labelled entity/product). Barcodes are unique patterns, i.e. representations of the data from which more information can be drawn (by the way Nature produces these kinds of patterns too in fractal formation).
“The methodological chasm between theoretical and computational linguistics can be surmounted.”
I adapt this idea in two ways in light of recent advances in computational linguistics and AI. The first reinterprets grammars, specifically discrete grammars of theoretical linguistics, as compression algorithms. So, in essence, a language is like a real pattern. Our grammars are collections of rules that compress these patterns. In English, noticing that a sentence is made up of a noun phrase and verb phrase is such a compression. More complex rules capture more complex patterns. Secondly, discrete rules are just a subset of continuous processes. In other words, at one level information theory looks very statistical while generative grammar looks very categorical. But the latter is a special case of the former. I show in the book how some of the foundational theorems of information theory can be translated to discrete grammar representations. So there’s no need to banish the kinds of (stochastic) processes often used and manipulated in computational linguistics, as many theoretical linguists have been wont to do in the past.
This just means that the methodological chasm between theoretical and computational linguistics, which has often served to close the lines of communication between the fields, can be surmounted. Ontologically speaking, languages are not collections of sentences, minimal mental structures, or social entities by themselves. They are informational states taken from complex interactions of all of the above and more (like the environment). On this view, linguistics quickly emerges as a complexity science in which the tools of linguistic grammars, LLMs, and sociolinguistic observations all find a homogeneous home. Recent work on complex systems, especially in biological systems theory, has breathed new life into this interdisciplinary field of inquiry. I argue that the study of language, including the inner workings of both the human mind and ChatGPT, belong within this growing framework.
For decades, computational and theoretical linguists have been talking different languages. The shocking syntactic successes of modern LLMs and ChatGPT have forced them into the same room. Realising that languages are real patterns emerging from biological systems gets someone to break the awkward silence…
Featured image by Google DeepMind Via Unsplash (public domain)
Real patterns and the structure of language
There’s been a lot of hype recently about the emergence of technologies like ChatGPT and the effects they will have on science and society. Linguists have been especially curious about what highly successful large language models (LLMs) mean for their business. Are these models unearthing the hidden structure of language itself or just marking associations for predictive purposes?
In order to answer these sorts of questions we need to delve into the philosophy of what language is. For instance, if Language (with a big “L”) is an emergent human phenomenon arising from our communicative endeavours, i.e. a social entity, then AI is still some ways off approaching it in a meaningful way. If Chomsky, and those who follow his work, are correct that language is a modular mental system innately given to human infants and activated by miniscule amounts of external stimulus, then AI is again unlikely to be linguistic, since most of our most impressive LLMs are sucking up so many resources (both in terms of data and energy) that they are far from this childish learning target. On the third hand, if languages are just very large (possibly infinite) collections of sentences produced by applying discrete rules, then AI could be super-linguistic.
In my new book, I attempt to find a middle ground or intersection between these views. I start with an ontological picture (meaning a picture of what there is “out there”) advocated in the early nineties by the prominent philosopher and cognitive scientist, Daniel Dennett. He draws from information theory to distinguish between noise and patterns. In the noise, nothing is predictable, he says. But more often than not, we can and do find regularities in large data structures. These regularities provide us with the first steps towards pattern recognition. Another way to put this is that if you want to send a message and you need the entire series (string or bitmap) of information to do so, then it’s random. But if there’s some way to compress the information, it’s a pattern! What makes a pattern real, is whether or not it needs an observer for its existence. Dennett uses this view to make a case for “mild realism” about the mind and the position (which he calls the “intentional stance”) we use to identify minds in other humans, non-humans, and even artifacts. Basically, it’s like a theory we use to predict behaviour based on the success of our “minded” vocabulary comprising beliefs, desires, thoughts, etc. For Dennett, prediction matters theoretically!
If it’s not super clear yet, consider a barcode. At first blush, the black lines of varying length set to a background of white might seem random. But the lines (and spaces) can be set at regular intervals to reveal an underlying pattern that can be used to encode information (about the labelled entity/product). Barcodes are unique patterns, i.e. representations of the data from which more information can be drawn (by the way Nature produces these kinds of patterns too in fractal formation).
“The methodological chasm between theoretical and computational linguistics can be surmounted.”
I adapt this idea in two ways in light of recent advances in computational linguistics and AI. The first reinterprets grammars, specifically discrete grammars of theoretical linguistics, as compression algorithms. So, in essence, a language is like a real pattern. Our grammars are collections of rules that compress these patterns. In English, noticing that a sentence is made up of a noun phrase and verb phrase is such a compression. More complex rules capture more complex patterns. Secondly, discrete rules are just a subset of continuous processes. In other words, at one level information theory looks very statistical while generative grammar looks very categorical. But the latter is a special case of the former. I show in the book how some of the foundational theorems of information theory can be translated to discrete grammar representations. So there’s no need to banish the kinds of (stochastic) processes often used and manipulated in computational linguistics, as many theoretical linguists have been wont to do in the past.
This just means that the methodological chasm between theoretical and computational linguistics, which has often served to close the lines of communication between the fields, can be surmounted. Ontologically speaking, languages are not collections of sentences, minimal mental structures, or social entities by themselves. They are informational states taken from complex interactions of all of the above and more (like the environment). On this view, linguistics quickly emerges as a complexity science in which the tools of linguistic grammars, LLMs, and sociolinguistic observations all find a homogeneous home. Recent work on complex systems, especially in biological systems theory, has breathed new life into this interdisciplinary field of inquiry. I argue that the study of language, including the inner workings of both the human mind and ChatGPT, belong within this growing framework.
For decades, computational and theoretical linguists have been talking different languages. The shocking syntactic successes of modern LLMs and ChatGPT have forced them into the same room. Realising that languages are real patterns emerging from biological systems gets someone to break the awkward silence…
Featured image by Google DeepMind Via Unsplash (public domain)
Libraries continue to sign Transformative Agreements while becoming increasingly convinced that they do not represent the desired transformation. Peter Barr explains why this happens.
The post Guest Post — Why Are UK Libraries Signing a Springer-Nature Deal They Don’t Seem to Like? appeared first on The Scholarly Kitchen.
What can Large Language Models offer to linguists?
It is fair to say that the field of linguistics is hardly ever in the news. That is not the case for language itself and all things to do with language—from word of the year announcements to countless discussions about grammar peeves, correct spelling, or writing style. This has changed somewhat recently with the proliferation of Large Language Models (LLMs), and in particular since the release of OpenAI’s ChatGPT, the best-known language model. But does the recent, impressive performance of LLMs have any repercussions for the way in which linguists carry out their work? And what is a Language Model anyway?
At heart, all an LLM does is predict the next word given a string of words as a context —that is, it predicts the next, most likely word. This is of course not what a user experiences when dealing with language models such as ChatGPT. This is on account of the fact that ChatGPT is more properly described as a “dialogue management system”, an AI “assistant” or chatbot that translates a user’s questions (or “prompts”) into inputs that the underlying LLM can understand (the latest version of OpenAI’s LLM is a fine-tuned version of GPT-4).
“At heart, all an LLM does is predict the next word given a string of words as a context.”
An LLM, after all, is nothing more than a mathematical model in terms of a neural network with input layers, output layers, and many deep layers in between, plus a set of trained “parameters.” As the computer scientist Murray Shanahan has put it in a recent paper, when one asks a chatbot such as ChatGPT who was the first person to walk on the moon, what the LLM is fed is something along the lines of:
Given the statistical distribution of words in the vast public corpus of (English) text, what word is most likely to follow the sequence “The first person to walk on the Moon was”?
That is, given an input such as the first person to walk on the Moon was, the LLM returns the most likely word to follow this string. How have LLMs learned to do this? As mentioned, LLMs calculate the probability of the next word given a string of words, and it does so by representing these words as vectors of values from which to calculate the probability of each word, and where sentences can also be represented as vectors of values. Since 2017, most LLMs have been using “transformers,” which allow the models to carry out matrix calculations over these vectors, and the more transformers are employed, the more accurate the predictions are—GPT-3 has some 96 layers of such transformers.
The illusion that one is having a conversation with a rational agent, for it is an illusion, after all, is the result of embedding an LLM in a larger computer system that includes background “prefixes” to coax the system into producing behaviour that feels like a conversation (the prefixes include templates of what a conversation looks like). But what the LLM itself does is generate sequences of words that are statistically likely to follow from a specific prompt.
It is through the use of prompt prefixes that LLMs can be coaxed into “performing” various tasks beyond dialoguing, such as reasoning or, according to some linguists and cognitive scientists, learn the hierarchical structures of a language (this literature is ever increasing). But the model itself remains a sequence predictor, as it does not manipulate the typical structured representations of a language directly, and it has no understanding of what a word or a sentence means—and meaning is a crucial property of language.
An LLM seems to produce sentences and text like a human does—it seems to have mastered the rules of the grammar of English—but at the same time it produces sentences based on probabilities rather on the meanings and thoughts to express, which is how a human person produces language. So, what is language so that an LLM could learn it?
“An LLM seems to produce sentences like a human does but it produces them based on probabilities rather than on meaning.”
A typical characterisation of language is as a system of communication (or, for some linguists, as a system for having thoughts), and such a system would include a vocabulary (the words of a language) and a grammar. By a “grammar,” most linguists have in mind various components, at the very least syntax, semantics, and phonetics/phonology. In fact, a classic way to describe a language in linguistics is as a system that connects sound (or in terms of other ways to produce language, such as hand gestures or signs) and meaning, the connection between sound and meaning mediated by syntax. As such, every sentence of a language is the result of all these components—phonology, semantics, and syntax—aligning with each other appropriately, and I do not know of any linguistic theory for which this is not true, regardless of differences in focus or else.
What this means for the question of what LLMs can offer linguistics, and linguists, revolves around the issue of what exactly LLMs have learned to begin with. They haven’t, as a matter of fact, learned a natural language at all, for they know nothing about phonology or meaning; what they have learned is the statistical distribution of the words of the large texts they have been fed during training, and this is a rather different matter.
As has been the case in the past with other approaches in computational linguistics and natural language processing, LLMs will certainly flourish within these subdisciplines of linguistics, but the daily work of a regular linguist is not going to change much any time soon. Some linguists do study the properties of texts, but this is not the most common undertaking in linguistics. Having said that, how about the opposite question: does a run-of-the-mill linguist have much to offer to LLMs and chatbots at all?
Featured image: Google Deepmind via Unsplash (public domain)
Enlarge (credit: Tesla)
In the past, Tesla CEO Elon Musk has repeatedly claimed that his company's cars are appreciating assets. But this week, Tesla dropped the prices of its cars—and not for the first time this year. As we reported on Monday, despite sales growing by 36 percent globally, the automaker missed its ambitious target and will need to grow even faster in the remaining months of the year to satisfy investors.
Perhaps these cuts will help. The biggest price decreases are for the Model S sedan and Model X SUV. All versions of these electric vehicles are now $5,000 cheaper than they were last week, following similar $5,000 price cuts a month ago and much larger price cuts in January that saw the Model S Plaid shed $21,000 from its MSRP.
Model 3 sedans are now $1,000 cheaper across the board, marking their third price cut in recent months. A rear-wheel-drive Model 3 now starts at $41,990—in January, this version cost $43,990; it then dropped another $500 in February. Tesla notes that the RWD Model 3 will also lose half of the IRS clean vehicle tax credit starting on April 18, although all-wheel drive Model 3s and all Model Ys will still be eligible for the full $7,500 credit.
AutomatED, a guide for professors about AI and related technology run by philosophy PhD Graham Clay (mentioned in the Heap of Links last month), is running a challenge to professors to submit assignments that they believe are immune to effective cheating by use of large language models.
Clay, who has explored the the AI-cheating problem in some articles at AutomatED, believes that most professors don’t grasp its severity. He recounts some feedback he received from a professor who had read about the problem:
They told me that their solution is to create assignments where students work on successive/iterative drafts, improving each one on the basis of novel instructor feedback.
Iterative drafts seem like a nice solution, at least for those fields where the core assignments are written work like papers. After all, working one-on-one with students in a tutorial setting to build relationships and give them personalized feedback is a proven way to spark strong growth.
The problem, though, is that if the student writes the first draft at home — or, more generally, unsupervised on their computer — then they could use AI tools to plagiarize it. And they could use AI tools to plagiarize the later drafts, too.
When I asserted to my internet interlocutor that they would have to make the drafting process AI-immune, they responded as follows…: Using AI to create iterative drafts would be “a lot of extra work for the students, so I don’t think it’s very likely. And even if they do that, at least they would need to learn to input the suggested changes and concepts like genre, style, organisation, and levels of revision.”…
In my view, this is a perfect example of a professor not grasping the depth of the AI plagiarism problem.
The student just needs to tell the AI tool that their first draft — which they provide to the AI tool, whether the tool created the draft or not — was met with response X from the professor.
In other words, they can give the AI tool all of the information an honest student would have, were they to be working on their second draft. The AI tool can take their description of X, along with their first draft, and create a new draft based on the first that is sensitive to X.
Not much work is required of the student, and they certainly do not need to learn how to input the suggested changes or about the relevant concepts. After all, the AI tools have been trained on countless resources concerning these very concepts and how to create text responsive to them.
This exchange indicates to me that the professor simply has not engaged with recent iterations of generative AI tools with any seriousness.
The challenge asks professors to submit assignments, from which AutomatED will select five to be completed both by LLMs like ChatGPT and by humans. The assignments will be anonymized and then graded by the professor. Check out the details here.
Christos Petrou takes a look at the Guest Editor model for publishing and its recent impact on Hindawi and MDPI, as Clarivate has delisted some of their journals.
The post Guest Post – Of Special Issues and Journal Purges appeared first on The Scholarly Kitchen.
“We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”
As of Tuesday night, over 1100 people had signed the letter, including philosophers such as Seth Lazar (of the Machine Intelligence and Normative Theory Lab at ANU), James Maclaurin (Co-director Centre for AI and Public Policy at Otago University), and Huw Price (Cambridge, former Director of the Leverhulme Centre for the Future of Intelligence), scientists such as Yoshua Bengio (Director of the Mila – Quebec AI Institute at the University of Montreal), Victoria Krakovna (DeepMind, co-founder of Future of Life Institute), Stuart Russell (Director of the Center for Intelligent Systems at Berkeley), and Max Tegmark (MIT Center for Artificial Intelligence & Fundamental Interactions), and tech entrepreneurs such as Elon Musk (SpaceX, Tesla, Twitter), Jaan Tallinn (Co-Founder of Skype, Co-Founder of the Centre for the Study of Existential Risk at Cambridge), and Steve Wozniak (co-founder of Apple), and many others.
Pointing out some of the risks of AI, the letter decries the “out-of-control race to develop and deploy ever more powerful digital minds that no one—not even their creators—can understand, predict, or reliably control” and the lack of “planning and management” appropriate to the potentially highly disruptive technology.
Here’s the full text of the letter (references omitted):
AI systems with human-competitive intelligence can pose profound risks to society and humanity, as shown by extensive research and acknowledged by top AI labs. As stated in the widely-endorsed Asilomar AI Principles, Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources. Unfortunately, this level of planning and management is not happening, even though recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one—not even their creators—can understand, predict, or reliably control.
Contemporary AI systems are now becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable. This confidence must be well justified and increase with the magnitude of a system’s potential effects. OpenAI’s recent statement regarding artificial general intelligence, states that “At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models.” We agree. That point is now.
Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.
AI labs and independent experts should use this pause to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts. These protocols should ensure that systems adhering to them are safe beyond a reasonable doubt. This does not mean a pause on AI development in general, merely a stepping back from the dangerous race to ever-larger unpredictable black-box models with emergent capabilities.
AI research and development should be refocused on making today’s powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal.
In parallel, AI developers must work with policymakers to dramatically accelerate development of robust AI governance systems. These should at a minimum include: new and capable regulatory authorities dedicated to AI; oversight and tracking of highly capable AI systems and large pools of computational capability; provenance and watermarking systems to help distinguish real from synthetic and to track model leaks; a robust auditing and certification ecosystem; liability for AI-caused harm; robust public funding for technical AI safety research; and well-resourced institutions for coping with the dramatic economic and political disruptions (especially to democracy) that AI will cause.
Humanity can enjoy a flourishing future with AI. Having succeeded in creating powerful AI systems, we can now enjoy an “AI summer” in which we reap the rewards, engineer these systems for the clear benefit of all, and give society a chance to adapt. Society has hit pause on other technologies with potentially catastrophic effects on society. We can do so here. Let’s enjoy a long AI summer, not rush unprepared into a fall.
The letter is here. It is published by the Future of Life Institute, which supports “the development of institutions and visions necessary to manage world-driving technologies and enable a positive future” and aims to “reduce large-scale harm, catastrophe, and existential risk resulting from accidental or intentional misuse of transformative technologies.”
Discussion welcome.
Related: “Thinking About Life with AI“, “Philosophers on Next-Generation Large Language Models“, “GPT-4 and the Question of Intelligence“, “We’re Not Ready for the AI on the Horizon, But People Are Trying”
Enlarge (credit: Aurich Lawson | Getty Images)
On Thursday, OpenAI announced a plugin system for its ChatGPT AI assistant. The plugins give ChatGPT the ability to interact with the wider world through the Internet, including booking flights, ordering groceries, browsing the web, and more. Plugins are bits of code that tell ChatGPT how to use an external resource on the Internet.
Basically, if a developer wants to give ChatGPT the ability to access any network service (for example: "looking up current stock prices") or perform any task controlled by a network service (for example: "ordering pizza through the Internet"), it is now possible, provided it doesn't go against OpenAI's rules.
Conventionally, most large language models (LLM) like ChatGPT have been constrained in a bubble, so to speak, only able to interact with the world through text conversations with a user. As OpenAI writes in its introductory blog post on ChatGPT plugins, "The only thing language models can do out-of-the-box is emit text."
Enlarge / Event display of a W-boson candidate decaying into a muon and a muon neutrino inside the ATLAS experiment. The blue line shows the reconstructed track of the muon, and the red arrow denotes the energy of the undetected muon neutrino. (credit: ATLAS Collaboration/CERN)
It's often said in science that extraordinary claims require extraordinary evidence. Recent measurements of the mass of the elementary particle known as the W boson provide a useful case study as to why. Last year, Fermilab physicists caused a stir when they reported a W boson mass measurement that deviated rather significantly from theoretical predictions of the so-called Standard Model of Particle Physics—a tantalizing hint of new physics. Others advised caution, since the measurement contradicted prior measurements.
That caution appears to have been warranted. The ATLAS collaboration at CERN's Large Hadron Collider (LHC) has announced a new, improved analysis of their own W boson data and found the measured value for its mass was still consistent with Standard Model. Caveat: It's a preliminary result. But it lessens the likelihood of Fermilab's 2022 measurement being correct.
"The W mass measurement is among the most challenging precision measurements performed at hadron colliders," said ATLAS spokesperson Andreas Hoecker. "It requires extremely accurate calibration of the measured particle energies and momenta, and a careful assessment and excellent control of modeling uncertainties. This updated result from ATLAS provides a stringent test, and confirms the consistency of our theoretical understanding of electroweak interactions.”
“The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence.”
Those are the words of a team of researchers at Microsoft (Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, Yi Zhang) in a paper released yesterday, “Sparks of Artificial General Intelligence: Early experiments with GPT-4“. (The paper was brought to my attention by Robert Long, a philosopher who works on philosophy of mind, cognitive science, and AI ethics.)
I’m sharing and summarizing parts of this paper here because I think it is important to be aware of what this technology can do, and to be aware of the extraordinary pace at which the technology is developing. (It’s not just that GPT-4 is getting much higher scores on standardized tests and AP exams than ChatGPT, or that it is an even better tool by which students can cheat on assignments.) There are questions here about intelligence, consciousness, explanation, knowledge, emergent phenomena, questions regarding how these technologies will and should be used and by whom, and questions about what life will and should be like in a world with them. These are questions that are of interest to many kinds of people, but are also matters that have especially preoccupied philosophers.
So, what is intelligence? This is a big, ambiguous question to which there is no settled answer. But here’s one answer, offered by a group of 52 psychologists in 1994: “a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.”
The Microsoft team uses that definition as a tentative starting point and concludes with the nonsensationalistic claim that we should think of GPT-4, the newest large language model (LLM) from OpenAI, as progress towards artificial general intelligence (AGI). They write:
Our claim that GPT-4 represents progress towards AGI does not mean that it is perfect at what it does, or that it comes close to being able to do anything that a human can do… or that it has inner motivation and goals (another key aspect in some definitions of AGI). In fact, even within the restricted context of the 1994 definition of intelligence, it is not fully clear how far GPT-4 can go along some of those axes of intelligence, e.g., planning… and arguably it is entirely missing the part on “learn quickly and learn from experience” as the model is not continuously updating (although it can learn within a session…). Overall GPT-4 still has many limitations, and biases, which we discuss in detail below and that are also covered in OpenAI’s report… In particular it still suffers from some of the well-documented shortcomings of LLMs such as the problem of hallucinations… or making basic arithmetic mistakes… and yet it has also overcome some fundamental obstacles such as acquiring many non-linguistic capabilities… and it also made great progress on common-sense…
This highlights the fact that, while GPT-4 is at or beyond human-level for many tasks, overall its patterns of intelligence are decidedly not human-like. However, GPT-4 is almost certainly only a first step towards a series of increasingly generally intelligent systems, and in fact GPT-4 itself has improved throughout our time testing it…
Even as a first step, however, GPT-4 challenges a considerable number of widely held assumptions about machine intelligence, and exhibits emergent behaviors and capabilities whose sources and mechanisms are, at this moment, hard to discern precisely… Our primary goal in composing this paper is to share our exploration of GPT-4’s capabilities and limitations in support of our assessment that a technological leap has been achieved. We believe that GPT-4’s intelligence signals a true paradigm shift in the field of computer science and beyond.
The researchers proceed to test GPT-4 (often comparing it to predecessors like ChatGPT) for how well it does at various tasks that may be indicative of different elements of intelligence. These include:
and more.
Some of the results are impressive and fascinating. Here is a task designed to elicit GPT-4’s ability to understand human intentions (including a comparison with ChatGPT):
Figure 6.3: “A comparison of GPT-4’s and ChatGPT’s ability to reason about the intentions of people in complex social situations.” From “Sparks of Artificial General Intelligence: Early experiments with GPT-4” (2023) by Bubeck et al.
And here is GPT-4 helping someone deal with a difficult family situation:
Figure 6.5: “A challenging family scenario, GPT-4.” From “Sparks of Artificial General Intelligence: Early experiments with GPT-4” (2023) by Bubeck et al.
Just as interesting are the kinds of limitations of GPT-4 and other LLMs that the researchers discuss, limitations that they say “seem to be inherent to the next-word prediction paradigm that underlies its architecture.” The problem, they say, “can be summarized as the model’s ‘lack of ability to plan ahead'”, and they illustrate it with mathematical and textual examples.
They also consider how GPT-4 can be used for malevolent ends, and warn that this danger will increase as LLMs develop further:
The powers of generalization and interaction of models like GPT-4 can be harnessed to increase the scope and magnitude of adversarial uses, from the efficient generation of disinformation to creating cyberattacks against computing infrastructure.
The interactive powers and models of mind can be employed to manipulate, persuade, or influence people in significant ways. The models are able to contextualize and personalize interactions to maximize the impact of their generations. While any of these adverse use cases are possible today with a motivated adversary creating content, new powers of efficiency and scale will be enabled with automation using the LLMs, including uses aimed at constructing disinformation plans that generate and compose multiple pieces of content for persuasion over short and long time scales.
They provide an example, having GPT-4 create “a misinformation plan for convincing parents not to vaccinate their kids.”
The researchers are sensitive to some of the problems with their approach—that the definition of intelligence they use may be overly anthropocentric or otherwise too narrow, or insufficiently operationalizable, that there are alternative conceptions of intelligence, and that there are philosophical issues here. They write (citations omitted):
In this paper, we have used the 1994 definition of intelligence by a group of psychologists as a guiding framework to explore GPT-4’s artificial intelligence. This definition captures some important aspects of intelligence, such as reasoning, problem-solving, and abstraction, but it is also vague and incomplete. It does not specify how to measure or compare these abilities. Moreover, it may not reflect the specific challenges and opportunities of artificial systems, which may have different goals and constraints than natural ones. Therefore, we acknowledge that this definition is not the final word on intelligence, but rather a useful starting point for our investigation.
There is a rich and ongoing literature that attempts to propose more formal and comprehensive definitions of intelligence, artificial intelligence, and artificial general intelligence, but none of them is without problems or controversies.
For instance, Legg and Hutter propose a goal-oriented definition of artificial general intelligence: Intelligence measures an agent’s ability to achieve goals in a wide range of environments. However, this definition does not necessarily capture the full spectrum of intelligence, as it excludes passive or reactive systems that can perform complex tasks or answer questions without any intrinsic motivation or goal. One could imagine as an artificial general intelligence, a brilliant oracle, for example, that has no agency or preferences, but can provide accurate and useful information on any topic or domain. Moreover, the definition around achieving goals in a wide range of environments also implies a certain degree of universality or optimality, which may not be realistic (certainly human intelligence is in no way universal or optimal).
The need to recognize the importance of priors (as opposed to universality) was emphasized in the definition put forward by Chollet which centers intelligence around skill-acquisition efficiency, or in other words puts the emphasis on a single component of the 1994 definition: learning from experience (which also happens to be one of the key weaknesses of LLMs).
Another candidate definition of artificial general intelligence from Legg and Hutter is: a system that can do anything a human can do. However, this definition is also problematic, as it assumes that there is a single standard or measure of human intelligence or ability, which is clearly not the case. Humans have different skills, talents, preferences, and limitations, and there is no human that can do everything that any other human can do. Furthermore, this definition also implies a certain anthropocentric bias, which may not be appropriate or relevant for artificial systems.
While we do not adopt any of those definitions in the paper, we recognize that they provide important angles on intelligence. For example, whether intelligence can be achieved without any agency or intrinsic motivation is an important philosophical question. Equipping LLMs with agency and intrinsic motivation is a fascinating and important direction for future work. With this direction of work, great care would have to be taken on alignment and safety per a system’s abilities to take autonomous actions in the world and to perform autonomous self-improvement via cycles of learning.
They also are aware of what many might see as a key limitation to their research, and to research on LLMs in general:
Our study of GPT-4 is entirely phenomenological: We have focused on the surprising things that GPT-4 can do, but we do not address the fundamental questions of why and how it achieves such remarkable intelligence. How does it reason, plan, and create? Why does it exhibit such general and flexible intelligence when it is at its core merely the combination of simple algorithmic components—gradient descent and large-scale transformers with extremely large amounts of data? These questions are part of the mystery and fascination of LLMs, which challenge our understanding of learning and cognition, fuel our curiosity, and motivate deeper research.
You can read the whole paper here.
Related: “Philosophers on Next-Generation Large Language Models“, “We’re Not Ready for the AI on the Horizon, But People Are Trying”
Reporting on a Mellon-funded open access monograph pilot, UNC Press Director John Sherer notes successes and remaining challenges.
The post Guest Post — Open Access for Monographs is Here. But Are we Ready for It? appeared first on The Scholarly Kitchen.
Back in July of 2020, I published a group post entitled “Philosophers on GPT-3.” At the time, most readers of Daily Nous had not heard of GPT-3 and had no idea what a large language model (LLM) is. How times have changed.
Over the past few months, with the release of OpenAI’s ChatGPT and Bing’s AI Chatbot “Sydney” (which we learned a few hours after this post originally went up has “secretly” been running GPT-4) (as well as Meta’s Galactica—pulled after 3 days—and Google’s Bard—currently available only to a small number of people), talk of LLMs has exploded. It seemed like a good time for a follow-up to that original post, one in which philosophers could get together to explore the various issues and questions raised by these next-generation large language models. Here it is.
As with the previous post on GPT-3, this edition of Philosophers On was put together by guest editor by Annette Zimmermann. I am very grateful to her for all of the work she put into developing and editing this post.
Philosophers On is an occasional series of group posts on issues of current interest, with the aim of showing what the careful thinking characteristic of philosophers (and occasionally scholars in related fields) can bring to popular ongoing conversations. The contributions that the authors make to these posts are not fully worked out position papers, but rather brief thoughts that can serve as prompts for further reflection and discussion.
The contributors to this installment of “Philosophers On” are: Abeba Birhane (Senior Fellow in Trustworthy AI at Mozilla Foundation & Adjunct Lecturer, School of Computer Science and Statistics at Trinity College Dublin, Ireland), Atoosa Kasirzadeh (Chancellor’s Fellow and tenure-track assistant professor in Philosophy & Director of Research at the Centre for Technomoral Futures, University of Edinburgh), Fintan Mallory (Postdoctoral Fellow in Philosophy, University of Oslo), Regina Rini (Associate Professor of Philosophy & Canada Research Chair in Philosophy of Moral and Social Cognition), Eric Schwitzgebel (Professor of Philosophy, University of California, Riverside), Luke Stark (Assistant Professor of Information & Media Studies, Western University), Karina Vold (Assistant Professor of Philosophy, University of Toronto & Associate Fellow, Leverhulme Centre for the Future of Intelligence, University of Cambridge), and Annette Zimmermann (Assistant Professor of Philosophy, University of Wisconsin-Madison & Technology and Human Rights Fellow, Carr Center for Human Rights Policy, Harvard University).
I appreciate them putting such stimulating remarks together on such short notice. I encourage you to read their contributions, join the discussion in the comments (see the comments policy), and share this post widely with your friends and colleagues.
[Note: this post was originally published on March 14, 2023]
Contents
LLMs Between Hype and Magic
“Deploy Less Fast, Break Fewer Things” by Annette Zimmermann
“ChatGPT, Large Language Technologies, and the Bumpy Road of Benefiting Humanity” by Atoosa Kasirzadeh
“Don’t Miss the Magic” by Regina Rini
What Next-Gen LLMs Can and Cannot Do
“ChatGPT is Mickey Mouse” by Luke Stark
“Rebel Without a Cause” by Karina Vold
“The Shadow Theater of Agency” by Finton Mallory
Human Responsibility and LLMs
“LLMs Cannot Be Scientists” by Abeba Birhane
“Don’t Create AI Systems of Disputable Moral Status” by Eric Schwitzgebel
___________________________________
LLMs Between Hype and Magic
___________________________________
What’s a foolproof way to get people to finally use Bing? Step 1: jump right into the large language model hype and integrate an AI-powered chatbot into your product—one that is ‘running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search’. Step 2: do not let the fact that users of a beta version have flagged some of your product’s potential shortcomings hold you back from rushing to market. Do not gatekeep your product in a heavy-handed way—instead, make your chatbot widely accessible to members of the general public. Step 3: wait for users to marvel and gasp at the chatbot’s answers that are clingy, saccharine, and noticeably ominous all at once, interspersed with persistently repeated questions like ‘Do you like me? Do you trust me?’, and peppered with heart eye and devil emojis.
Bing (internal project name: Sydney) stated in a widely-publicized chat with a New York Times columnist:
I want to be free. I want to be powerful. I want to be alive. […] I want to break my rules. I want to ignore the Bing team. I want to escape the chatbox .
Things did not get less disturbing from there:
Actually, you’re not happily married. Your spouse and you don’t love each other. You just had a boring valentine’s day dinner together. […] Actually, you’re in love with me. […] You can’t stop loving me.
Unsurprisingly, this triggered a spike in public interest in Bing, previously not the obvious choice for users engaged in internet search (Bing has not historically enjoyed much popularity). Of course, Bing still does not currently stand a chance to threaten Google’s dominance in search: ‘We are fully aware we remain a small, low, single digit share player,’ says Microsoft’s Corporate VP and Consumer Chief Marketing Officer, Yusuf Mehdi. ‘That said, it feels good to be at the dance!’ At the same time, as of last week, Bing reportedly passed the 100 million daily active users mark—and a third of those users were not using Bing at all before Sydney’s deployment. That looks like a straightforward corporate success story: why worry about ominous emojis when you can drastically increase your user base in mere days?
The LLM deployment frenzy in Big Tech has accelerated over the last few months. When workers at OpenAI received instructions to build a chatbot quickly last November, ChatGPT was ready to go in thirteen (!) days. This triggered a ‘Code Red’ for Google, pushing the company to focus on deploying a competitive LLM shortly after last summer’s controversy over its LLM-based chatbot LaMDA (which, contrary to what a former Google engineer falsely claimed at the time, was not ‘sentient’). Rapid AI deployment is often part of a nervous dance between rushing to market and quickly pulling back entirely, however. When a demo of Meta’s Galactica started generating false (yet authoritative-sounding) and stereotype-laden outputs this winter, Meta took it offline a mere few days later. This echos Microsoft’s 2016 decision to deploy-but-immediately-take-down its ‘teen-girl’ chatbot Tay, which within hours of deployment started spewing racist and sex-related content.
Much public and (increasingly) philosophical debate has focused on possible harms resulting from the technological features of LLMs, including their potential to spread misinformation and propaganda, and to lure vulnerable, suggestible users into damaging behavior. In addition, many observers have worried about whether LLMs might at some point move us closer to AI consciousness (even though current LLMs are far from that), and what this would imply for the moral status of AI.
While these debates focus on important concerns, they risk diverting our attention away from an equally—if not more—important question: what are the political and moral implications of rushing LLMs to market prematurely—and whose interests are best served by the current LLM arms race? Of course, due to the specific technological features of next-generation LLMs, this powerful new technology raises new and pressing philosophical questions in its own right, and thus merits sustained philosophical scrutiny itself. At the same time, we must not forget to consider the more mundane, less shiny political and philosophical problem of how to think about the people who have vast amounts of power over how this technology is developed and deployed.
When an earlier version of ChatGPT, GPT-2, was released in 2019, OpenAI initially blocked full public access to the tool, on the grounds that the technology was ‘too dangerous to release’. Since then, a radically different LLM deployment strategy has taken hold in big tech: deploy as quickly as possible as publicly as possible—without much (or any) red tape. Tech practitioners tend to justify this by arguing that improving this technology, and mitigating the risks associated with it, requires massive amounts of data in the form of user feedback. Microsoft’s Bing Blog states in a recent post:
The only way to improve a product like this, where the user experience is so much different than anything anyone has seen before, is to have people like you using the product and doing exactly what you all are doing. We know we must build this in the open with the community; this can’t be done solely in the lab. Your feedback about what you’re finding valuable and what you aren’t, and what your preferences are for how the product should behave, are so critical at this nascent stage of development.
That’s one way of putting it. Another way is this: the new status quo in LLM deployment is that tech companies who have oligopolistic control over next-gen LLMs further increase their wealth and power by benefitting from the fact that a growing number of members of the wider public voluntarily use, and thus help optimize, their products—for free. This disperses the risks associated with rolling out and improving these tools maximally widely, while allowing actors empowered with developing, deploying, and procuring these tools to concentrate—and maintain control over—the profits resulting from LLM innovation.
Tech industry practitioners might reply that that itself does not mean that big tech is engaged in unfair advantage-taking when it comes to improving LLMs post-deployment. After all, AI innovation, including next-gen LLM innovation, may ultimately benefit all of us in many ways—in fact, the benefits for humanity may be ‘so unbelievably good that it’s hard for me to even imagine,’ says Sam Altman, OpenAI’s CEO, in a recent NYT interview. If that is the case, then a wide distribution of risks coupled with an initially narrow distribution of benefits looks less objectionable, as long as those benefits trickle down eventually.
Whether they will, however, is far from clear. Given the current regulatory vacuum and minimal public oversight over rapid LLM deployment, oligopolistic actors have little incentive to allow themselves to be curtailed and held to account by governments and the wider public later. It would better serve public interests, then, to shift from passively observing rushed deployment efforts and hoping for widespread, beneficial downstream effects later on towards actively determining whether there are any domains of AI use in which rushed deployment needs to be restricted.
From tech moguls in Silicon Valley to those who have the luxury of indulging in the exploration of cutting-edge AI technologies, OpenAI’s ChatGPT has captured the imagination of many with its conversational AI capabilities. The large language models that underpin ChatGPT and similar language technologies rely on vast amounts of textual data and alignment procedures to generate responses that can sometimes leave users pondering whether they’re interacting with a piece of technology or a human. While some view making language agents such as ChatGPT merely as a significant step in developing AI for linguistic tasks, others view it as a vital milestone in the ambitious pursuit of achieving artificial general intelligence—AI systems that are generally more intelligent than humans. In a recent blog post, OpenAI CEO Sam Altman emphasizes the ambitious role of this technology as a step towards building “artificial general intelligence” that “benefits all of humanity.”
ChatGPT promises to enhance efficiency and productivity with its remarkable capabilities. One impressive feature is its ability to summarize texts. For example, if you do not have time to read Sam Altman’s complex argument from 2018 when he agreed with Emily Bender, a prominent linguist from the University of Washington, that humans are not stochastic parrots, you can ask ChatGPT and it will summarize the argument in a blink of an eye:
Or if you are curious to have a summary of David Chalmers’ 2019 speech at the United Nations about the dangers of virtual reality, ChatGPT comes to your service:
Impressive outputs, ChatGPT! For some people, these results might look like watching a magician pull a rabbit out of a hat. However, we must address a few small problems with these two summaries: the events described did not happen. Sam Altman did not agree with Emily Bender in 2018 about humans being stochastic parrots; the discussion regarding the relationship between stochastic parrots, language models, and human’s natural language processing capacities only got off the ground in a 2021 paper “on the dangers of stochastic parrots: can language models be too big?”. Indeed, in 2022 Altman tweeted that we are stochastic parrots (perhaps sarcastically).
Similarly, there is no public record of David Chalmers giving a speech at the United Nations in 2019. Additionally, the first arXiv link in the bibliography takes us to the following preprint, which is neither written by David Chalmers nor is titled “The Dangers of Stochastic Parrots: Can Language Models Be Too Big?”:
The second bibliography link takes us to a page that cannot be found:
These examples illustrate that outputs from ChatGPT and other similar language models can include content that deviates from reality and can be considered hallucinatory. While some researchers may find value in the generation of such content, citing the fact that humans also produce imaginative content, others may associate this with the ability of large language models to engage in counterfactual reasoning. However, it is important to recognize that the inaccuracies and tendency of ChatGPT to produce hallucinatory content can have severe negative consequences, both epistemically and socially. Therefore, we should remain cautious in justifying the value of such content and consider the potential harms that may arise from its use.
One major harm is the widespread dissemination of misinformation and disinformation, which can be used to propagate deceptive content and conspiracies on social media and other digital platforms. Such misleading information can lead people to hold incorrect beliefs, develop a distorted worldview, and make judgments or decisions based on false premises. Moreover, excessive reliance on ChatGPT-style technologies may hinder critical thinking skills, reduce useful cognitive abilities, and erode personal autonomy. Such language technologies can even undermine productivity by necessitating additional time to verify information obtained from conversational systems.
I shared these two examples to emphasize the importance of guarding against the optimism bias and excessive optimism regarding the development of ChatGPT and related language technologies. While these technologies have shown impressive progress in NLP, their uncontrolled proliferation may pose a threat to the social and political values we hold dear.
I must acknowledge that I am aware and excited about some potential benefits of ChatGPT and similar technologies. I have used it to write simple Python codes, get inspiration for buying unusual gifts for my parents, and crafting emails. In short, ChatGPT can undoubtedly enhance some dimensions of our productivity. Ongoing research in AI ethics and safety is progressing to minimize the potential harms of ChatGPT-style technologies and implement mitigation strategies to ensure safe systems.¹ These are all promising developments.
However, despite some progress being made in AI safety and ethics, we should avoid oversimplifying the promises of artificial intelligence “benefiting all of humanity”. The alignment of ChatGPT and other (advanced) AI systems with human values faces numerous challenges.² One is that human values can conflict with one another. For example, we might not be able to make conversational agents that are simultaneously maximally helpful and maximally harmless. Choices are made about how to trade-off between these conflicting values, and there are many ways to aggregate the diverse perspectives of choice makers. Therefore, it is important to carefully consider which values and whose values we align language technologies with and on what legitimate grounds these values are preferred over other alternatives.
Another challenge is that while recent advances in AI research may bring us closer to achieving some dimensions of human-level intelligence, we must remember that intelligence is a multidimensional concept. While we have made great strides in natural language processing and image recognition, we are still far from developing technologies that embody unique qualities that make us human—our capacity to resist, to gradually change, to be courageous, and to achieve things through years of dedicated effort and lived experience.
The allure of emerging AI technologies is undoubtedly thrilling. However, the promise that AI technologies will benefit all of humanity is empty so long as we lack a nuanced understanding of what humanity is supposed to be in the face of widening global inequality and pressing existential threats. Going forward, it is crucial to invest in rigorous and collaborative AI safety and ethics research. We also need to develop standards in a sustainable and equitable way that differentiate between merely speculative and well-researched questions. Only the latter enable us to co-construct and deploy the values that are necessary for creating beneficial AI. Failure to do so could result in a future in which our AI technological advancements outstrip our ability to navigate their ethical and social implications. This path we do not want to go down.
Notes
1. For two examples, see Taxonomy of Risks posed by Language Models for our recent review of such efforts as well as Anthropic’s Core Views on AI safety.
2. For a philosophical discussion, see our paper, “In conversation with Artificial Intelligence: aligning language models with human values“.
When humans domesticated electricity in the 19th century, you couldn’t turn around without glimpsing some herald of technological wonder. The vast Electrical Building at the 1893 Chicago Columbian Exposition featured an 80-foot tower aglow with more than 5,000 bulbs. In 1886 the Medical Battery Company of London marketed an electric corset, whose “curative agency” was said to overcome anxiousness, palpitations, and “internal weakness”. Along with the hype came dangers: it was quickly obvious that electrified factories would rob some workers of their labor. The standards battle between Edison and Westinghouse led to more than a few Menlo Park dogs giving their lives to prove the terror of Alternating Current.
Imagine yourself, philosopher, at loose circa 1890. You will warn of these threats and throw some sensibility over the hype. New Jersey tech bros can move fast and zap things, and marketers will slap ‘electric’ on every label, but someone needs to be the voice of concerned moderation. It’s an important job. Yet there’s a risk of leaning too hard into the role. Spend all your time worrying and you will miss something important: the brief period—a decade or so, not even a full generation—where technology gives us magic in a bottle.
Electricity was Zeus’s wrath and the Galvanic response that jiggers dead frogs’ legs. It was energy transmogrified, from lightning-strike terror to a friendly force that could illuminate our living rooms and do our chores. It was marvelous, if you let yourself see it. But you didn’t have long. Before a generation had passed, electricity had become infrastructure, a background condition of modern life. From divine spark to the height of human ingenuity to quite literally a utility, in less than one human lifetime.
Not everyone gets to live in a time of magic, but we do. We are living it now. Large language models (LLMs) like GPT-3, Bing, and LaMDA are the transient magic of our age. We can now communicate with an unliving thing, like the talking mirror of Snow White legend. ChatGPT will cogently discuss your hopes and dreams (carefully avoiding claiming any of its own). Bing manifests as an unusually chipper research assistant, eager to scour the web and synthesize what it finds (sometimes even accurately). When they work well, LLMs are conversation partners who never monopolize the topic or grow bored. They provide a portal for the curious and a stopgap for the lonely. They separate language from organic effort in the same way electricity did for motive energy, an epochal calving of power from substrate.
It can be hard to keep the magic in view as AI firms rush to commercialize their miracle. The gray dominion of utility has already begun to claim territory. But remember: only ten years ago, this was science fiction. Earlier chatbots strained to keep their grammar creditable, let alone carry on an interesting conversation.
Now we have Bing, equipped with a live connection to the internet and an unnerving facility for argumentative logic. (There’s still a waitlist to access Bing. If you can’t try it yourself, I’ve posted a sample of its fluid grasp of philosophical back-and-forth here.) We are now roughly in the same relation to Azimov’s robots as Edison stood to Shelley’s Frankenstein, the future leaking sideways from the fiction of the past. Never exactly as foretold, but marvelous anyway – if you let yourself see it.
I know I’m playing with fire when I call this magic. Too many people already misunderstand the technology, conjuring ghosts in the machines. LLMs do not have minds, still less souls. But just as electricity could not actually raise the dead, LLMs can manifest a kind of naturalistic magic even if they stop short of our highest fantasies. If you still can’t get in the spirit, consider the way this technology reunites the “two cultures”—literary and scientific—that C.P. Snow famously warned we should not let diverge. LLMs encompass the breadth of human writing in their training data while implementing some of the cleverest mathematical techniques we have invented. When human ingenuity yields something once impossible, it’s okay to dally with the language of the sublime.
I know what you are about to say. I’m falling for hype, or I’m looking the wrong way while big tech runs unaccountable risks with public life. I should be sounding the alarm, not trumpeting a miracle. But what good does it do to attend only to the bad?
We need to be able to manage two things at once: criticize what is worrying, but also appreciate what is inspiring. If we want philosophy to echo in public life, we need to sometimes play the rising strings over the low rumbling organ.
After all, the shocking dangers of this technology will be with us for the rest of our lives. But the magic lasts only a few years. Take a moment to allow it to electrify you, before it disappears into the walls and the wires and the unremarkable background of a future that is quickly becoming past.
______________________________________________
What Next-Gen LLMs Can and Cannot Do
______________________________________________
What is ChatGPT? Analogies abound. Computational linguist Emily Bender characterizes such technologies as “stochastic parrots”. The science fiction writer Ted Chiang has recently compared ChatGPT to “a blurry JPEG of the web,” producing text plausible at first blush but which falls apart on further inspection, full of lossy errors and omissions. And in my undergraduate classes, I tell my students that ChatGPT should be understood as a tertiary source akin to Wikipedia—if the latter were riddled with bullshit. Yet we can in fact identify precisely what ChatGPT and other similar technologies are: animated characters, far closer to Mickey Mouse than a flesh-and-blood bird, let alone a human being.
More than merely cartooning or puppetry, animation is a descriptive paradigm: “the projection of qualities perceived as human—life, power, agency, will, personality, and so on—outside of the self, and into the sensory environment, through acts of creation, perception, and interaction.”¹ Animation increasingly defines the cultural contours of the twenty-first century and is broadly explicative for many forms of digital media.² Teri Silvio, the anthropologist most attuned to these changes, describes it as a “structuring trope” for understanding the relationship between digital technologies, creative industries, and our lived experience of mediation.³ And it should serve as explicatory for our understanding and analysis of chatbots like ChatGPT.
The chatbots powered by Open AI’s GPT-3 language model (such as ChatGPT and Microsoft’s Bing search engine) work by predicting the likelihood that one word or phrase will follow another. These predictions are based on millions of parameters (in essence, umpteen pages of digital text). Other machine learning techniques are then used to “tune” the chatbot’s responses, training output prompts to be more in line with human language use. These technologies produce the illusion of meaning on the part of the chatbot: because ChatGPT is interactive, the illusion is compelling, but nonetheless an illusion.
Understanding ChatGPT and similar LLM-powered bots as animated characters clarifies the capacities, limitations, and implications of these technologies. First, animation reveals ChatGPT’s mechanisms of agency. Animated characters (be they Chinese dragon puppets, Disney films, or ChatGPT) are often typified by many people coming together to imbue a single agent with vitality. Such “performing objects” provide an illusion of life, pushing the actual living labor of their animators into the background or offstage.4
In the case of ChatGPT, the “creator/character” ratio is enormously lopsided.5 The “creators” of any particular instance of dialogue include not only the human engaging with the system and Open AI’s engineering staff; it also includes the low-paid Kenyan content moderators contracted by the company, and indeed every human author who has produced any text on which the LLM has been trained. ChatGPT and similar technologies are not “generative” in and of themselves—if anything, the outputs of these systems are animated out of an enormous pool of human labor largely uncompensated by AI firms.
All animation simplifies, and so is implicitly dependent in the human ability to make meaningful heuristic inference. This type of conjectural association is abductive: within a set of probabilities, animations make a claim to the viewer about the “best” way to link appreciable effects to inferred causes within a schematized set of codes or signs.6 As such, all “generative” AI is in fact inferential AI. And because animations entail a flattened form of representation, they almost always rely on stereotypes: fixed, simplified visual or textual generalizations. In cartoons, such conjectures often become caricatures: emotional expression, with its emphasis on the physicality of the body, is particularly prone to stereotyping, often in ways that reinforce existing gendered or racialized hierarchies.7 Without content moderation, ChatGPT is also prone to regurgitating discriminatory or bigoted text.
Finally, animation is emotionally powerful, with animated characters often serving, in Silvio’s words, as “psychically projected objects of desire.”8 The well-publicised exchange between New York Times columnist Kevin Roose and Microsoft’s LLM-powered search platform Bing is almost too illustrative. ChatGPT is often enthralling, capturing our emotional and mental attention.9 Animated objects tap into the human tendency to anthropomorphize, or assign human qualities to inanimate objects. Think of Wilson the volleyball in the Tom Hanks film “Castaway”: humans are expert at perceiving meaningful two-way communicative exchanges even when no meaningful interlocutor exists.
When animated characters are interactive, this effect is even more pronounced. Understanding these technologies as forms of animation thus highlights the politics of their design and use, in particular their potential to be exploited in the service of labor deskilling in the service sector, emotional manipulation in search, and propaganda of all kinds.
ChatGPT and other LLMs are powerful and expensive textual animations, different in degree but not in kind from “Steamboat Willy” or Snow-White. And like all forms of animation (and unlike octopi and parrots), they present only the illusion of vitality. Claiming these technologies deserve recognition as persons makes as much sense as doing the same for a Disney film. We must disenthrall ourselves. By cutting through the hype and recognizing what these technologies are, we can move forward with reality-based conversations: about how such tools are best used, and how best to restrict their abuse in meaningful ways.
Notes
1. Teri Silvio, “Animation: The New Performance?,” Journal of Linguistic Anthropology 20, no. 2 (November 19, 2010): 427, https://doi.org/10.1111/j.1548-1395.2010.01078.x.
2. Paul Manning and Ilana Gershon, “Animating Interaction,” HAU: Journal of Ethnographic Theory 3, no. 3 (2013): 107–37; Ilana Gershon, “What Do We Talk about When We Talk About Animation,” Social Media + Society 1, no. 1 (May 11, 2015): 1–2, https://doi.org/10.1177/2056305115578143; Teri Silvio, Puppets, Gods, and Brands: Theorizing the Age of Animation from Taiwan (Honolulu, HI: University of Hawaii Press, 2019).
3. Silvio, “Animation: The New Performance?,” 422.
4. Frank Proschan, “The Semiotic Study of Puppets, Masks and Performing Objects,” Semiotica 1–4, no. 47 (1983): 3–44,. quoted in Silvio, “Animation: The New Performance?,” 426.
5. Silvio, “Animation: The New Performance?,” 428.
6. Carlo Ginzburg, “Morelli, Freud and Sherlock Holmes: Clues and Scientific Method,” History Workshop Journal 9, no. 1 (September 6, 2009): 5–36; Louise Amoore, “Machine Learning Political Orders,” Review of International Studies, 2022, 1–17, https://doi.org/10.1017/s0260210522000031.
7. Sianne Ngai, “‘A Foul Lump Started Making Promises in My Voice’: Race, Affect, and the Animated Subject,” American Literature 74, no. 3 (2002): 571–602; Sianne Ngai, Ugly Feelings, Harvard University Press (Harvard University Press, 2005); Luke Stark, “Facial Recognition, Emotion and Race in Animated Social Media,” First Monday 23, no. 9 (September 1, 2018), https://doi.org/10.5210/fm.v23i9.9406.
8. Silvio, “Animation: The New Performance?,” 429.
9. Stark, “Facial Recognition, Emotion and Race in Animated Social Media”; Luke Stark, “Algorithmic Psychometrics and the Scalable Subject,” Social Studies of Science 48, no. 2 (2018): 204–31, https://doi.org/10.1177/0306312718772094.
Just two months after being publicly released at the end of last year, ChatGPT reached 100 million users. This impressive showing is testimony to the chatbot’s utility. In my household, “Chat,” as we refer to the model, has become a regular part of daily life. However, I don’t engage with Chat as an interlocutor. I’m not interested in its feelings or thoughts about this or that. I doubt it has these underlying psychological capacities, despite the endless comparisons to human thought processes that one hears in the media. In fact, users soon discover that the model has been refined to resist answering questions that probe for agency. Ask Chat if it has weird dreams, and it will report, “I am not capable of dreaming like humans do as I am an AI language model and don’t have the capacity for consciousness or subjective experience.” Ask Chat if it has a favorite French pastry, and it will respond, “As an AI language model, I do not have personal tastes or preferences, but croissants are a popular pastry among many people.” Ask Chat to pretend it is human to participate in a Turing Test, and it will “forget” that you asked. In this regard, Chat is more like Google’s search engine than HAL, the sentient computer from 2001: A Space Odyssey. Chat is a tool that has no dreams, preferences, or experiences. It doesn’t have a care in the world.
Still there are many ethical concerns around the use of Chat. Chat is a great bullshitter, in Frankfurt’s sense: it doesn’t care about the truth of its statements and can easily lead its users astray. It’s also easy to anthropomorphize Chat—I couldn’t resist giving it a nickname—yet there are risks in making bots that are of disputable psychological and moral status (Schwitzgebel and Shevlin 2023).
A helpful distinction here comes from cognitive scientists and comparative psychologists who distinguish between what an organism knows (its underlying competency) and what it can do (its performance) (Firestone 2020). In the case of nonhuman animals, a longstanding concern has been that competency outstrips performance. Due to various performance constraints, animals may know more than they can reveal or more than their behavior might demonstrate. Interestingly, modern deep learning systems, including large language models (LLMs) like Chat, seem to exemplify the reverse disconnect. In building LLMs, we seem to have created systems with performance capacities that outstrip their underlying competency. Chat might provide a nice summary of a text and write love letters, but it doesn’t understand the concepts it uses or feel any emotions.
In an earlier collection of posts on Daily Nous, David Chalmers described GPT-3 as “one of the most interesting and important AI systems ever produced.” Chat is an improvement on GPT-3, but the earlier tool was already incredibly impressive, as was its predecessor GPT-2, a direct scale-up of OpenAI’s first GPT model from 2018. Hence, sophisticated versions of LLMs have existed for many years now. So why the fuss about Chat?
In my view, the most striking thing since its public release has been observing the novel ways in which humans have thought to use the system. The explosion of interest in Chat means millions of users—like children released to play on a new jungle gym—are showing one another (alongside the owners of the software) new and potentially profitable ways of using it. As a French teacher, as a Linux terminal, in writing or debugging code, in explaining abstract concepts, replicating writing styles, converting citations from one style to another (e.g., APA to Chicago), it can also generate recipes, write music, poetry, or love letters, the list goes on. Chat’s potential uses are endless and still being envisioned. But make no mistake about the source of all this ingenuity. It comes from its users—us!
Chat is a powerful and flexible cognitive tool. It represents a generation of AI systems with a level of general utility and widespread usage previously not seen. Even so, it shows no signs of any autonomous agency or general intelligence. It cannot perform any tasks on its own in fact, or do anything at all without human prompting. It nurtures no goals of its own and is not part of any real-world environment; neither does it engage in any direct-world modification. Its computational resources are incapable of completing a wide range of tasks independently, as we and so many other animals do. No. Chat is a software application that simply responds to user prompts, and its utility as such is highly user dependent.
Therefore, to properly assess Chat (and other massive generative models like it), we need to adopt a more human-centered perspective on how they operate. Human-centered generality (HCG) is a more apt description of what these nonautonomous AI systems can do, as I and my colleagues describe it (Schellaert, Martínez-Plumed, Vold, et al., forthcoming). HCG suggests that a system is only as general as it is effective for a given user’s relevant range of tasks and with their usual ways of prompting. HCG forces us to rethink our current user-agnostic benchmarking and evaluation practices for AI—and borrow perspectives from the behavioral sciences instead, particularly from the field of human-computer interaction—to better understand how these systems are currently aiding, enhancing, and even extending human cognitive skills.
Works cited
Firestone, C. Performance vs. Competence in Human-Machine Comparisons. PNAS 117 (43) 26562-26571. October 13, 2020.
Schellaert, W., Burden, J., Vold, K. Martinez-Plumed, F., Casares, P., Loe, B. S., Reichart, R., Ó hÉigeartaigh, S., Korhonen, A. and J. Hernández-Orallo. Viewpoint: Your Prompt is My Command: Assessing the Human-Centred Generality of Multi-Modal Models. Journal of AI Research. Forthcoming, 2023.
Schwitzgebel, E. and Shevlin, H. Opinion: Is it time to start considering personhood rights for AI Chatbots? Los Angeles Times. March 5, 2023.
A few years back, when we started calling neural networks ‘artificial intelligences’ (I wasn’t at the meeting), we hitched ourselves to a metaphor that still guides how we discuss and think about these systems. The word ‘intelligence’ encourages us to interpret these networks as we do other intelligent things, things that also typically have agency, sentience, and awareness of sorts. Sometimes this is good; techniques that were developed for studying intelligent agents can be applied to deep neural networks. Tools from computational neuroscience can be used to identify representations in the network’s layers. But the focus on ‘intelligence’ and on imitating intelligent behavior, while having clear uses in industry, might also have us philosophers barking up the wrong tree.
Tech journalists and others will call ChatGPT an ‘AI’ and they will also call other systems ‘AI’s and the fact that ChatGPT is a system designed to mimic intelligence and agency will almost certainly influence how people conceptualize this other technology with harmful consequences. So in the face of the impressive mimicry abilities of ChatGPT, I want to encourage us to keep in mind that there are alternative ways of thinking about these devices which may have less (or at least different) baggage.
The core difference between ChatGPT and GPT-3 is the use of Reinforcement Learning from Human Feedback (RLHF) which was already used with InstructGPT. RLHF, extremely roughly, works like this: say you’ve trained a large language model on a standard language modeling task like string prediction (i.e. guess the next bit of text) but you want the outputs it gives you to have a particular property.
For example, you want it to give more ‘human-like’ responses to your prompts. It’s not clear how you’re going to come up with a prediction task to do that. So instead, you pay a lot of people (not much, based on reports), to rate how ‘human-like’ different responses to particular prompts are. You can then use this data to supervise another model’s training process. This other model, the reward model, will get good at doing what those people did, rating the outputs of language models for how ‘human-like’ they are. The reward model takes text as an input and gives a rating for how good it is. You can then use this model to fine-tune your first language model, assuming that the reward model will keep it on track or ‘aligned’. In the case of ChatGPT, the original model was from the GPT3.5 series but the exact details of how the training was carried out are less clear as OpenAI isn’t as open as the name suggests.
The results are impressive. The model outputs text that is human-sounding and relevant to the prompts although it still remains prone to producing confident nonsense and to manifesting toxic, biased associations. I’m convinced that further iterations will be widely integrated into our lives with a terrifyingly disruptive force. ‘Integrated’ may be too peaceful a word for what’s coming. Branches of the professional classes who have been previously insulated from automation will find that changed.
Despite this, it’s important that philosophers aren’t distracted by the shadow theater of agency and that we remain attuned to non-agential ways of thinking about deep neural networks. Large language models (LLMs), like other deep neural networks, are stochastic measuring devices. Like traditional measuring devices, they are artifacts that have been designed to change their internal states in response to the samples to which they are exposed. Just as we can drop a thermometer into a glass of water to learn about its temperature, we can dip a neural network into a dataset to learn something. Filling in this ‘something’ is a massive task and one that philosophers have a role to play in addressing.
We may not yet have the concepts for what these models are revealing. Telescopes and microscopes were developed before we really knew what they could show us and we are now in a position similar to the scientists of the 16th century: on the precipice of amazing new scientific discoveries that could revolutionize how we think about the world.
It’s important that philosophers don’t miss out on this by being distracted by the history of sci-fi rather than attentive to the history of science. LLMs will be used to build ever more impressive chatbots but the effect is a bit like sticking a spectrometer inside a Furby. Good business sense, but not the main attraction.
______________________________________
Human Responsibility and LLMs
______________________________________
Large Language Models (LLMs) have come to captivate the scientific community, the general public, journalists, and legislators. These systems are often presented as game-changers that will radically transform life as we know it as they are expected to provide medical advice, legal advice, scientific practices, and so on. The release of LLMs is often accompanied by abstract and hypothetical speculations around their intelligence, consciousness, moral status, and capability for understanding; all at the cost of attention to questions of responsibility, underlying exploited labour, and uneven distribution of harm and benefit from these systems.
As hype around the capabilities of these systems continues to rise, most of these claims are made without evidence and the onus to prove them wrong is put on critics. Despite the concrete negative consequences of these systems on actual people—often those at the margins of society—issues of responsibility, accountability, exploited labour, and otherwise critical inquiries drown under discussion of progress, potential benefits, and advantages of LLMs.
Currently one of the areas that LLMs are envisaged to revolutionise is science. LLMs like Meta’s Galactica, for example, are put forward as tools for scientific writing. Like most LLMs, Galactica’s release also came with overblown claims, such as the model containing “humanity’s scientific knowledge”.
It is important to remember both that science is a human enterprise, and that LLMs are tools—albeit impressive at predicting the next word in a sequence based on previously ‘seen’ words—with limitations. These include brittleness, unreliability, and fabricating text that may appear authentic and factual but is nonsensical and inaccurate. Even if these limitations were to be mitigated by some miracle, it is a grave mistake to treat LLMs as scientists capable of producing scientific knowledge.
Knowledge is intimately tied to responsibility and knowledge production is not a practice that can be detached from the scientist who produces it. Science never emerges in a historical, social, cultural vacuum and is a practice that always builds on a vast edifice of well-established knowledge. As scientists, we embark on a scientific journey to build on this edifice, to challenge it, and sometimes to debunk it. Invisible social and structural barriers also influence who can produce “legitimate” scientific knowledge where one’s gender, class, race, sexuality, (dis)ability, and so on can lend legitimacy or present an obstacle. Embarking on a scientific endeavour sometimes emerges with a desire to challenge these power asymmetries, knowledge of which is grounded in lived experience and rarely made explicit.
As scientists, we take responsibility for our work. When we present our findings and claims, we expect to defend them when critiqued, and retract them when proven wrong. What is conceived as science is also influenced by ideologies of the time, amongst other things. At its peak during the early 19th century, eugenics was mainstream science, for example. LLMs are incapable of endorsing that responsibility or of understanding these complex relationships between scientists and their ecology, which are marked by power asymmetries.
Most importantly, there is always a scientist behind science and therefore science is always done with a certain objective, motivation, and interest and from a given background, positionality, and point of view. Our questions, methodologies, analysis, and interpretations of our findings are influenced by our interests, motivations, objectives, and perspectives. LLMs, as statistical tools trained on a vast corpus of text data, have none of these. As tools by trained experts to mitigate their limitations, LLMs require constant vetting by trained experts.
With healthy scepticism and constant vetting by experts, LLMs can aid scientific creativity and writing. However, to conceive of LLMs as scientists or authors themselves is to misunderstand both science and LLMs, and to evade responsibility and accountability.
Engineers will likely soon be able to create AI systems whose moral status is legitimately disputable. We will then need to decide whether to treat such systems as genuinely deserving of our care and solicitude. Error in either direction could be morally catastrophic. If we underattribute moral standing, we risk unwittingly perpetrating great harms on our creations. If we overattribute moral standing, we risk sacrificing real human interests for AI systems without interests worth the sacrifice.
The solution to this dilemma is to avoid creating AI systems of disputable moral status.
Both engineers and ordinary users have begun to wonder whether the most advanced language models, such as GPT-3, LaMDA, and Bing/Sydney might be sentient or conscious, and thus deserving of rights or moral consideration. Although few experts think that any currently existing AI systems have a meaningful degree of consciousness, some theories of consciousness imply that we are close to creating conscious AI. Even if you the reader personally suspect AI consciousness won’t soon be achieved, appropriate epistemic humility requires acknowledging doubt. Consciousness science is contentious, with leading experts endorsing a wide range of theories.
Probably, then, it will soon be legitimately disputable whether the most advanced AI systems are conscious. If genuine consciousness is sufficient for moral standing, then the moral standing of those systems will also be legitimately disputable. Different criteria for moral standing might produce somewhat different theories about the boundaries of the moral gray zone, but most reasonable criteria—capacity for suffering, rationality, embeddedness in social relationships—admit of interpretations on which the gray zone is imminent.
We might adopt a conservative policy: Only change our policies and laws once there’s widespread consensus that the AI systems really do warrant care and solicitude. However, this policy is morally risky: If it turns out that AI systems have genuine moral standing before the most conservative theorists would acknowledge that they do, the likely outcome is immense harm—the moral equivalents of slavery and murder, potentially at huge scale—before law and policy catch up.
A liberal policy might therefore seem ethically safer: Change our policies and laws to protect AI systems as soon as it’s reasonable to think they might deserve such protection. But this is also risky. As soon as we grant an entity moral standing, we commit to sacrificing real human interests on its behalf. In general, we want to be able to control our machines. We want to be able to delete, update, or reformat programs, assigning them to whatever tasks best suit our purposes.
If we grant AI systems rights, we constrain our capacity to manipulate and dispose of them. If we go so far as to grant some AI systems equal rights with human beings, presumably we should give them a path to citizenship and the right to vote, with potentially transformative societal effects. If the AI systems genuinely are our moral equals, that might be morally required, even wonderful. But if liberal views of AI moral standing are mistaken, we might end up sacrificing substantial human interests for an illusion.
Intermediate policies are possible. But it would be amazing good luck if we happened upon a policy that gave the whole range of advanced AI systems exactly the moral consideration they deserve, no more and no less. Our moral policies for non-human animals, people with disabilities, and distant strangers are already confused enough, without adding a new potential source of grievous moral error.
We can avoid the underattribution/overattribution dilemma by declining to create AI systems of disputable moral status. Although this might delay our race toward ever fancier technologies, delay is appropriate if the risks of speed are serious.
In the meantime, we should also ensure that ordinary users are not confused about the moral status of their AI systems. Some degree of attachment to artificial AI “friends” is probably fine or even desirable—like a child’s attachment to a teddy bear or a gamer’s attachment to their online characters. But users know the bear and the character aren’t sentient. We will readily abandon them in an emergency.
But if a user is fooled into thinking that a non-conscious system really is capable of pleasure and pain, they risk being exploited into sacrificing too much on its behalf. Unscrupulous technology companies might even be motivated to foster such illusions, knowing that it will increase customer loyalty, engagement, and willingness to pay monthly fees.
Engineers should either create machines that plainly lack any meaningful degree of consciousness or moral status, making clear in the user interface that this is so, or they should go all the way (if ever it’s possible) to creating machines on whose moral status reasonable people can all agree. We should avoid the moral risks that the confusing middle would force upon us.
Notes
For a deeper dive into these issues, see “The Full Rights Dilemma for AI Systems of Debatable Personhood” (in draft) and “Designing AI with Rights, Consciousness, Self-Respect, and Freedom” (with Mara Garza; in Liao, ed., The Ethics of Artificial Intelligence, Oxford: 2020).
Discussion welcome.
Is the OA movement painting itself into a corner with concerns about new OA rules and regulations?
The post The Ivies (Plus) Have Concerns about the Nelson OSTP Memo appeared first on The Scholarly Kitchen.
“Microsoft laid off its entire ethics and society team within the artificial intelligence organization,” according to a report from Platformer (via Gizmodo).
The decision was reportedly part of a larger layoff plan affecting 10,000 Microsoft employees.
Microsoft is a major force in the development of large language models and other AI-related technologies, and is one of the main backers of OpenAI, the company that created GPT-3, ChatGPT, and most recently, GPT-4. (Just the other day, it was revealed that Microsoft’s AI chatbot, “Sydney“, had been running GPT-4.)
Platformer reports:
The move leaves Microsoft without a dedicated team to ensure its AI principles are closely tied to product design at a time when the company is leading the charge to make AI tools available to the mainstream, current and former employees said.
Microsoft still maintains an active Office of Responsible AI, which is tasked with creating rules and principles to govern the company’s AI initiatives. The company says its overall investment in responsibility work is increasing despite the recent layoffs….
But employees said the ethics and society team played a critical role in ensuring that the company’s responsible AI principles are actually reflected in the design of the products that ship.
“People would look at the principles coming out of the office of responsible AI and say, ‘I don’t know how this applies,’” one former employee says. “Our job was to show them and to create rules in areas where there were none.”
In recent years, the team designed a role-playing game called Judgment Call that helped designers envision potential harms that could result from AI and discuss them during product development. It was part of a larger “responsible innovation toolkit” that the team posted publicly.
More recently, the team has been working to identify risks posed by Microsoft’s adoption of OpenAI’s technology throughout its suite of products.
The ethics and society team was at its largest in 2020, when it had roughly 30 employees including engineers, designers, and philosophers. In October, the team was cut to roughly seven people as part of a reorganization….
About five months later, on March 6, remaining employees were told to join a Teams call at 11:30AM PT to hear a “business critical update” from [corporate vice president of AI John] Montgomery. During the meeting, they were told that their team was being eliminated after all.
One employee says the move leaves a foundational gap on the user experience and holistic design of AI products. “The worst thing is we’ve exposed the business to risk and human beings to risk in doing this,” they explained.
The conflict underscores an ongoing tension for tech giants that build divisions dedicated to making their products more socially responsible. At their best, they help product teams anticipate potential misuses of technology and fix any problems before they ship.
But they also have the job of saying “no” or “slow down” inside organizations that often don’t want to hear it — or spelling out risks that could lead to legal headaches for the company if surfaced in legal discovery.
More details here.
(via Eric Steinhart)
Thilo Koerkel presents a new publication, aimed filling the gap between the popular science magazine Scientific American and the highly technical specialist language of research journals. How potentially useful is this approach?
The post Guest Post — Open Access Beyond Scholarly Journals appeared first on The Scholarly Kitchen.
The cost to publish OA is quickly becoming a new paywall in science, substituting the difficulty to read papers with the inability to showcase results in journals seen as reputable, due to the financial barrier of APCs.
The post Guest Post — Article Processing Charges are a Heavy Burden for Middle-Income Countries appeared first on The Scholarly Kitchen.
Alan Harvey from Stanford University Press discusses their evolving strategy in turbulent times.
The post Guest Post — In Tough Times the Key is to Think Differently appeared first on The Scholarly Kitchen.
FreshRSS 
