Login
“There are good reasons to think that some AIs today have wellbeing.”
In this guest post, Simon Goldstein (Dianoia Institute, Australian Catholic University) and Cameron Domenico Kirk-Giannini (Rutgers University – Newark, Center for AI Safety) argue that some existing artificial intelligences have a kind of moral significance because they’re beings for whom things can go well or badly.
This is the sixth in a series of weekly guest posts by different authors at Daily Nous this summer.
[Posts in the summer guest series will remain pinned to the top of the page for the week in which they’re published.]
We recognize one another as beings for whom things can go well or badly, beings whose lives may be better or worse according to the balance they strike between goods and ills, pleasures and pains, desires satisfied and frustrated. In our more broad-minded moments, we are willing to extend the concept of wellbeing also to nonhuman animals, treating them as independent bearers of value whose interests we must consider in moral deliberation. But most people, and perhaps even most philosophers, would reject the idea that fully artificial systems, designed by human engineers and realized on computer hardware, may similarly demand our moral consideration. Even many who accept the possibility that humanoid androids in the distant future will have wellbeing would resist the idea that the same could be true of today’s AI.
Perhaps because the creation of artificial systems with wellbeing is assumed to be so far off, little philosophical attention has been devoted to the question of what such systems would have to be like. In this post, we suggest a surprising answer to this question: when one integrates leading theories of mental states like belief, desire, and pleasure with leading theories of wellbeing, one is confronted with the possibility that the technology already exists to create AI systems with wellbeing. We argue that a new type of AI—the artificial language agent—has wellbeing. Artificial language agents augment large language models with the capacity to observe, remember, and form plans. We also argue that the possession of wellbeing by language agents does not depend on them being phenomenally conscious. Far from a topic for speculative fiction or future generations of philosophers, then, AI wellbeing is a pressing issue. This post is a condensed version of our argument. To read the full version, click here.
Artificial language agents (or simply language agents) are our focus because they support the strongest case for wellbeing among existing AIs. Language agents are built by wrapping a large language model (LLM) in an architecture that supports long-term planning. An LLM is an artificial neural network designed to generate coherent text responses to text inputs (ChatGPT is the most famous example). The LLM at the center of a language agent is its cerebral cortex: it performs most of the agent’s cognitive processing tasks. In addition to the LLM, however, a language agent has files that record its beliefs, desires, plans, and observations as sentences of natural language. The language agent uses the LLM to form a plan of action based on its beliefs and desires. In this way, the cognitive architecture of language agents is familiar from folk psychology.
For concreteness, consider the language agents built this year by a team of researchers at Stanford and Google. Like video game characters, these agents live in a simulated world called ‘Smallville’, which they can observe and interact with via natural-language descriptions of what they see and how they act. Each agent is given a text backstory that defines their occupation, relationships, and goals. As they navigate the world of Smallville, their experiences are added to a “memory stream” in the form of natural language statements. Because each agent’s memory stream is long, agents use their LLM to assign importance scores to their memories and to determine which memories are relevant to their situation. Then the agents reflect: they query the LLM to make important generalizations about their values, relationships, and other higher-level representations. Finally, they plan: They feed important memories from each day into the LLM, which generates a plan for the next day. Plans determine how an agent acts, but can be revised on the fly on the basis of events that occur during the day. In this way, language agents engage in practical reasoning, deciding how to promote their goals given their beliefs.
The conclusion that language agents have beliefs and desires follows from many of the most popular theories of belief and desire, including versions of dispositionalism, interpretationism, and representationalism.
According to the dispositionalist, to believe or desire that something is the case is to possess a suitable suite of dispositions. According to ‘narrow’ dispositionalism, the relevant dispositions are behavioral and cognitive; ‘wide’ dispositionalism also includes dispositions to have phenomenal experiences. While wide dispositionalism is coherent, we set it aside here because it has been defended less frequently than narrow dispositionalism.
Consider belief. In the case of language agents, the best candidate for the state of believing a proposition is the state of having a sentence expressing that proposition written in the memory stream. This state is accompanied by the right kinds of verbal and nonverbal behavioral dispositions to count as a belief, and, given the functional architecture of the system, also the right kinds of cognitive dispositions. Similar remarks apply to desire.
According to the interpretationist, what it is to have beliefs and desires is for one’s behavior (verbal and nonverbal) to be interpretable as rational given those beliefs and desires. There is no in-principle problem with applying the methods of radical interpretation to the linguistic and nonlinguistic behavior of a language agent to determine what it believes and desires.
According to the representationalist, to believe or desire something is to have a mental representation with the appropriate causal powers and content. Representationalism deserves special emphasis because “probably the majority of contemporary philosophers of mind adhere to some form of representationalism about belief” (Schwitzgebel).
It is hard to resist the conclusion that language agents have beliefs and desires in the representationalist sense. The Stanford language agents, for example, have memories which consist of text files containing natural language sentences specifying what they have observed and what they want. Natural language sentences clearly have content, and the fact that a given sentence is in a given agent’s memory plays a direct causal role in shaping its behavior.
Many representationalists have argued that human cognition should be explained by positing a “language of thought.” Language agents also have a language of thought: their language of thought is English!
An example may help to show the force of our arguments. One of Stanford’s language agents had an initial description that included the goal of planning a Valentine’s Day party. This goal was entered into the agent’s planning module. The result was a complex pattern of behavior. The agent met with every resident of Smallville, inviting them to the party and asking them what kinds of activities they would like to include. The feedback was incorporated into the party planning.
To us, this kind of complex behavior clearly manifests a disposition to act in ways that would tend to bring about a successful Valentine’s Day party given the agent’s observations about the world around it. Moreover, the agent is ripe for interpretationist analysis. Their behavior would be very difficult to explain without referencing the goal of organizing a Valentine’s Day party. And, of course, the agent’s initial description contained a sentence with the content that its goal was to plan a Valentine’s Day party. So, whether one is attracted to narrow dispositionalism, interpretationism, or representationalism, we believe the kind of complex behavior exhibited by language agents is best explained by crediting them with beliefs and desires.
What makes someone’s life go better or worse for them? There are three main theories of wellbeing: hedonism, desire satisfactionism, and objective list theories. According to hedonism, an individual’s wellbeing is determined by the balance of pleasure and pain in their life. According to desire satisfactionism, an individual’s wellbeing is determined by the extent to which their desires are satisfied. According to objective list theories, an individual’s wellbeing is determined by their possession of objectively valuable things, including knowledge, reasoning, and achievements.
On hedonism, to determine whether language agents have wellbeing, we must determine whether they feel pleasure and pain. This in turn depends on the nature of pleasure and pain.
There are two main theories of pleasure and pain. According to phenomenal theories, pleasures are phenomenal states. For example, one phenomenal theory of pleasure is the distinctive feeling theory. The distinctive feeling theory says that there is a particular phenomenal experience of pleasure that is common to all pleasant activities. We see little reason why language agents would have representations with this kind of structure. So if this theory of pleasure were correct, then hedonism would predict that language agents do not have wellbeing.
The main alternative to phenomenal theories of pleasure is attitudinal theories. In fact, most philosophers of wellbeing favor attitudinal over phenomenal theories of pleasure (Bramble). One attitudinal theory is the desire-based theory: experiences are pleasant when they are desired. This kind of theory is motivated by the heterogeneity of pleasure: a wide range of disparate experiences are pleasant, including the warm relaxation of soaking in a hot tub, the taste of chocolate cake, and the challenge of completing a crossword. While differing in intrinsic character, all of these experiences are pleasant when desired.
If pleasures are desired experiences and AIs can have desires, it follows that AIs can have pleasure if they can have experiences. In this context, we are attracted to a proposal defended by Schroeder: an agent has a pleasurable experience when they perceive the world being a certain way, and they desire the world to be that way. Even if language agents don’t presently have such representations, it would be possible to modify their architecture to incorporate them. So some versions of hedonism are compatible with the idea that language agents could have wellbeing.
We turn now from hedonism to desire satisfaction theories. According to desire satisfaction theories, your life goes well to the extent that your desires are satisfied. We’ve already argued that language agents have desires. If that argument is right, then desire satisfaction theories seem to imply that language agents can have wellbeing.
According to objective list theories of wellbeing, a person’s life is good for them to the extent that it instantiates objective goods. Common components of objective list theories include friendship, art, reasoning, knowledge, and achievements. For reasons of space, we won’t address these theories in detail here. But the general moral is that once you admit that language agents possess beliefs and desires, it is hard not to grant them access to a wide range of activities that make for an objectively good life. Achievements, knowledge, artistic practices, and friendship are all caught up in the process of making plans on the basis of beliefs and desires.
Generalizing, if language agents have beliefs and desires, then most leading theories of wellbeing suggest that their desires matter morally.
We’ve argued that language agents have wellbeing. But there is a simple challenge to this proposal. First, language agents may not be phenomenally conscious — there may be nothing it feels like to be a language agent. Second, some philosophers accept:
The Consciousness Requirement. Phenomenal consciousness is necessary for having wellbeing.
The Consciousness Requirement might be motivated in either of two ways: First, it might be held that every welfare good itself requires phenomenal consciousness (this view is known as experientialism). Second, it might be held that though some welfare goods can be possessed by beings that lack phenomenal consciousness, such beings are nevertheless precluded from having wellbeing because phenomenal consciousness is necessary to have wellbeing.
We are not convinced. First, we consider it a live question whether language agents are or are not phenomenally conscious (see Chalmers for recent discussion). Much depends on what phenomenal consciousness is. Some theories of consciousness appeal to higher-order representations: you are conscious if you have appropriately structured mental states that represent other mental states. Sufficiently sophisticated language agents, and potentially many other artificial systems, will satisfy this condition. Other theories of consciousness appeal to a ‘global workspace’: an agent’s mental state is conscious when it is broadcast to a range of that agent’s cognitive systems. According to this theory, language agents will be conscious once their architecture includes representations that are broadcast widely. The memory stream of Stanford’s language agents may already satisfy this condition. If language agents are conscious, then the Consciousness Requirement does not pose a problem for our claim that they have wellbeing.
Second, we are not convinced of the Consciousness Requirement itself. We deny that consciousness is required for possessing every welfare good, and we deny that consciousness is required in order to have wellbeing.
With respect to the first issue, we build on a recent argument by Bradford, who notes that experientialism about welfare is rejected by the majority of philosophers of welfare. Cases of deception and hallucination suggest that your life can be very bad even when your experiences are very good. This has motivated desire satisfaction and objective list theories of wellbeing, which often allow that some welfare goods can be possessed independently of one’s experience. For example, desires can be satisfied, beliefs can be knowledge, and achievements can be achieved, all independently of experience.
Rejecting experientialism puts pressure on the Consciousness Requirement. If wellbeing can increase or decrease without conscious experience, why would consciousness be required for having wellbeing? After all, it seems natural to hold that the theory of wellbeing and the theory of welfare goods should fit together in a straightforward way:
Simple Connection. An individual can have wellbeing just in case it is capable of possessing one or more welfare goods.
Rejecting experientialism but maintaining Simple Connection yields a view incompatible with the Consciousness Requirement: the falsity of experientialism entails that some welfare goods can be possessed by non-conscious beings, and Simple Connection guarantees that such non-conscious beings will have wellbeing.
Advocates of the Consciousness Requirement who are not experientialists must reject Simple Connection and hold that consciousness is required to have wellbeing even if it is not required to possess particular welfare goods. We offer two arguments against this view.
First, leading theories of the nature of consciousness are implausible candidates for necessary conditions on wellbeing. For example, it is implausible that higher-order representations are required for wellbeing. Imagine an agent who has first order beliefs and desires, but does not have higher order representations. Why should this kind of agent not have wellbeing? Suppose that desire satisfaction contributes to wellbeing. Granted, since they don’t represent their beliefs and desires, they won’t themselves have opinions about whether their desires are satisfied. But the desires still are satisfied. Or consider global workspace theories of consciousness. Why should an agent’s degree of cognitive integration be relevant to whether their life can go better or worse?
Second, we think we can construct chains of cases where adding the relevant bit of consciousness would make no difference to wellbeing. Imagine an agent with the body and dispositional profile of an ordinary human being, but who is a ‘phenomenal zombie’ without any phenomenal experiences. Whether or not its desires are satisfied or its life instantiates various objective goods, defenders of the Consciousness Requirement must deny that this agent has wellbeing. But now imagine that this agent has a single persistent phenomenal experience of a homogenous white visual field. Adding consciousness to the phenomenal zombie has no intuitive effect on wellbeing: if its satisfied desires, achievements, and so forth did not contribute to its wellbeing before, the homogenous white field should make no difference. Nor is it enough for the consciousness to itself be something valuable: imagine that the phenomenal zombie always has a persistent phenomenal experience of mild pleasure. To our judgment, this should equally have no effect on whether the agent’s satisfied desires or possession of objective goods contribute to its wellbeing. Sprinkling pleasure on top of the functional profile of a human does not make the crucial difference. These observations suggest that whatever consciousness adds to wellbeing must be connected to individual welfare goods, rather than some extra condition required for wellbeing: rejecting Simple Connection is not well motivated. Thus the friend of the Consciousness Requirement cannot easily avoid the problems with experientialism by falling back on the idea that consciousness is a necessary condition for having wellbeing.
We’ve argued that there are good reasons to think that some AIs today have wellbeing. But our arguments are not conclusive. Still, we think that in the face of these arguments, it is reasonable to assign significant probability to the thesis that some AIs have wellbeing.
In the face of this moral uncertainty, how should we act? We propose extreme caution. Wellbeing is one of the core concepts of ethical theory. If AIs can have wellbeing, then they can be harmed, and this harm matters morally. Even if the probability that AIs have wellbeing is relatively low, we must think carefully before lowering the wellbeing of an AI without producing an offsetting benefit.
[Image made with DALL-E]
Some related posts:
“Philosophers on GPT-3”
“Philosophers on Next-Generation Large Language Models”
“GPT-4 and the Question of Intelligence”
“We’re Not Ready for the AI on the Horizon, But People Are Trying”
“Researchers Call for More Work on Consciousness”
“Dennett on AI: We Must Protect Ourselves Against ‘Counterfeit People’”
“Philosophy, AI, and Society Listserv”
“Talking Philosophy with Chat-GPT“
The post A Case for AI Wellbeing (guest post) first appeared on Daily Nous.
All 24 Russell Group universities have reviewed their academic conduct policies and guidance
UK universities have drawn up a set of guiding principles to ensure that students and staff are AI literate, as the sector struggles to adapt teaching and assessment methods to deal with the growing use of generative artificial intelligence.
Vice-chancellors at the 24 Russell Group research-intensive universities have signed up to the code. They say this will help universities to capitalise on the opportunities of AI while simultaneously protecting academic rigour and integrity in higher education.
Continue reading...
An AI door, according to a generative AI
Written by David Lyreskog
In what is quite possibly my last entry for the Practical Ethics blog, as I’m sadly leaving the Uehiro Centre in July, I would like to reflect on some things that have been stirring my mind the last year or so.
In particular, I have been thinking about thinking with machines, with people, and what the difference is.
–
The Uehiro Centre for Practical Ethics is located in an old carpet warehouse on an ordinary side street in Oxford. Facing the building, there is a gym to your left, and a pub to your right, mocking the researchers residing within the centre walls with a daily dilemma.
As you are granted access to the building – be it via buzzer or key card – a dry, somewhat sad, voice states “stay clear of the door” before the door slowly swings open.
–
The other day a colleague of mine shared a YouTube video of the presentation The AI Dilemma, by Tristan Harris and Aza Raskin. In it, they share with the audience their concerns about the rapid and somewhat wild development of artificial intelligence (AI) in the hands of a few tech giants. I highly recommend it. (The video, that is. Not the rapid and somewhat wild development of AI in the hands of a few tech giants).
Much like the thousands of signatories of the March open call to “pause giant AI experiments”, and recently the “Godfather of AI” Geoffrey Hinton, Harris and Raskin warn us that we are on the brink of major (negative, dangerous) social disruption due to the power of new AI technologies.
Indeed, there’s a bit of a public buzz about “AI ethics” in recent months.
While it is good that there is a general awareness and a public discussion about AI – or any majorly disruptive phenomenon for that matter – there’s a potential problem with the abstraction: AI is portrayed as this big, emerging, technological, behemoth which we cannot or will not control. But it has been almost three decades since humans were able to beat an AI at a game of chess. We have been using AI for many things, from medical diagnosis to climate prediction, with little to no concern about it besting us and/or stripping us of agency in these domains. In other domains, such as driving cars, and military applications of drones, there has been significantly more controversy.
All this is just to say that AI ethics is not for hedgehogs – it’s not “one big thing”[i] – and I believe that we need to actively avoid a narrative and a line of thinking which paints it to be. In examining the ethical dimensions of a multitude of AI inventions, then, we ought to take care to limit the scope of our inquiry to the domain in question at the very least.
So let us, for argument’s sake, return to that door at the Uehiro Centre, and the voice cautioning visitors to stay clear. Now, as far as I’m aware, the voice and the door are not part of an AI system. I also believe that there is no person who is tasked with waiting around for visitors asking for access, warning them of the impending door swing, and then manually opening the door. I believe it is a quite simple contraption, with a voice recording programmed to be played as the door opens. But does it make a difference to me, or other visitors, which of these possibilities is true?
We can call these possibilities:
Condition one (C1): AI door, created by humans.
Condition two (C2): Human speaker & door operator.
Condition three (C3): Automatic door & speaker, programmed by humans.
In C3, it seems that the outcome of the visitor’s action will always be the same after the buzzer is pushed or the key card is blipped: the voice will automatically say ‘stay clear of the door’, and the door will open. In C1 and C2, the same could be the case. But it could also be the case that the AI/human has been instructed to assess the risk for visitors on a case-to-case basis, and to only advise caution if there is imminent risk of collision or such (was this the case, I am consistently standing too close to the door when visiting, but that is beside the point).
On the surface, I think there are some key differences between these conditions which could have an ethical or moral impact, where some differences are more interesting than others. In C1 and C2, the door opener makes a real-time assessment, rather than following a predetermined cause of action in the way C3’s door opener does. More importantly, C2 is presumed to make this assessment from a place of concern, in a way which is impossible in C1 and C3 because the latter two are not moral agents, and therefore cannot be concerned. They simply do not have the capacity. And our inquiry could perhaps end here.
But it seems it would be a mistake.
What if something was to go wrong? Say the door swings open, but no voice warns me to stay clear, and so the door whacks me in the face[ii]. In C2, it seems the human who’s job it is to warn me of the imminent danger might have done something morally wrong, assuming they knew what to expect from opening the door without warning me, but failed in doing so due to negligence[iii]. In C1 and C3, on the other hand, while we may be upset with the door opener(s), we don’t believe that they did anything morally wrong – they just malfunctioned.
My colleague Alberto Giubilini highlighted the tensions in the morality of this landscape in what I thought was an excellent piece arguing that “It is not about AI, it is about humans”: we cannot trust AI, because trust is a relationship between moral agents, and AI does not (yet) have the capacity for moral agency and responsibility. We can, however, rely on AI to behave in a certain way (whether we should is a separate issue).
Similarly, while we may believe that a human should show concern for their fellow person, we should not expect the same from AIs, because they cannot be concerned.
Yet, if the automatic doors continue to whack visitors in the face, we may start feeling that someone should be responsible for this – not only legally, but morally: someone has a moral duty to ensure these doors are safe to pass through, right?
In doing so, we expand the field of inquiry, from the door opener to the programmer/constructor of the door opener, and perhaps to someone in charge of maintenance.
A couple of things pop to mind here.
First, when we find no immediate moral agent to hold responsible for a harmful event, we may expand the search field until we find one. That search seems to me to follow a systematic structure: if the door is automatic, we turn to call the support line, and if the support fails to fix the problem, but turns out to be an AI, we turn to whoever is in charge of support, and so on, until we find a moral agent.
Second, it seems to me that, if the door keeps slamming into visitors’ faces in condition in C2, we will not only morally blame the door operator, but also whoever left them in charge of that door. So perhaps the systems-thinking does not only apply when there is a lack of moral agents, but also applies on a more general level when we are de facto dealing with complicated and/or complex systems of agents.
Third, let us conjure a condition four (C4) like so: the door is automatic, but in charge of maintenance support is an AI system that is usually very reliable, and in charge of the AI support system, in turn, is a (human) person.
If the person in charge of an AI support system that failed to provide adequate service to a faulty automatic door is to blame for anything, it is plausibly for not adequately maintaining the AI support system – but not for whacking people in the face with a door (because they didn’t do that). Yet, perhaps there is some form of moral responsibility for the face-whacking to be found within the system as a whole. I.e. the compound of door-AI-human etc., has a moral duty to avoid face-whacking, regardless of any individual moral agents’ ability to whack faces.
If this is correct, it seems to me that we again[iv] find that our traditional means of ascribing moral responsibility fails to capture key aspects of moral life: it is not the case that any agent is individually morally responsible for the face-whacking door, nor are there multiple agents who are individually or collectively responsible for the face-whacking door. Yet, there seems to be moral responsibility for face-whacking doors in the system. Where does it come from, and what is its nature and structure (if it has one)?
In this way, not only cognitive processes such as thinking and computing seem to be able to be distributed throughout systems, but perhaps also moral capacities such as concern, accountability, and responsibility.
And in the end, I do not know to what extent it actually matters, at least in this specific domain. Because at the end of the day, I do not care much whether the door opener is human, an AI, or automatic.
I just need to know whether or not I need to stay clear of the door.
–
Notes & References.
[i] Berlin, I. (2013). The hedgehog and the fox: An essay on Tolstoy’s view of history. Princeton University Press.
[ii] I would like to emphasize that this is a completely hypothetical case, and that I take it to be safe to enter the Uehiro centre. The risk of face-whacking is, in my experience, minimal.
[iii] Let’s give them the benefit of the doubt here, and assume it wasn’t maleficence.
[iv] Together with Hazem Zohny, Julian Savulescu, and Ilina Singh, I have previously argued this to be the case in the domain of emerging technologies for collective thinking and decision-making, such as brain-to-brain interfaces. See the Open Access paper Merging Minds for more on this argument.
“Lying” in computer-generated texts: hallucinations and omissions
There is huge excitement about ChatGPT and other large generative language models that produce fluent and human-like texts in English and other human languages. But these models have one big drawback, which is that their texts can be factually incorrect (hallucination) and also leave out key information (omission).
In our chapter for The Oxford Handbook of Lying, we look at hallucinations, omissions, and other aspects of “lying” in computer-generated texts. We conclude that these problems are probably inevitable.
Omissions are inevitable because a computer system cannot cram all possibly-relevant information into a text that is short enough to be actually read. In the context of summarising medical information for doctors, for example, the computer system has access to a huge amount of patient data, but it does not know (and arguably cannot know) what will be most relevant to doctors.
Hallucinations are inevitable because of flaws in computer systems, regardless of the type of system. Systems which are explicitly programmed will suffer from software bugs (like all software systems). Systems which are trained on data, such as ChatGPT and other systems in the Deep Learning tradition, “hallucinate” even more. This happens for a variety of reasons. Perhaps most obviously, these systems suffer from flawed data (e.g., any system which learns from the Internet will be exposed to a lot of false information about vaccines, conspiracy theories, etc.). And even if a data-oriented system could be trained solely on bona fide texts that contain no falsehoods, its reliance on probabilistic methods will mean that word combinations that are very common on the Internet may also be produced in situations where they result in false information.
Suppose, for example, on the Internet, the word “coughing” is often followed by “… and sneezing.” Then a patient may be described falsely, by a data-oriented system, as “coughing and sneezing” in situations where they cough without sneezing. Problems of this kind are an important focus for researchers working on generative language models. Where this research will lead us is still uncertain; the best one can say is that we can try to reduce the impact of these issues, but we have no idea how to completely eliminate them.
“Large generative language models’ texts can be factually incorrect (hallucination) and leave out key information (omission).”
The above focuses on unintentional-but-unavoidable problems. There are also cases where a computer system arguably should hallucinate or omit information. An obvious example is generating marketing material, where omitting negative information about a product is expected. A more subtle example, which we have seen in our own work, is when information is potentially harmful and it is in users’ best interests to hide or distort it. For example, if a computer system is summarising information about sick babies for friends and family members, it probably should not tell an elderly grandmother with a heart condition that the baby may die, since this could trigger a heart attack.
Now that the factual accuracy of computer-generated text draws so much attention from society as a whole, the research community is starting to realize more clearly than before that we only have a limited understanding of what it means to speak the truth. In particular, we do not know how to measure the extent of (un)truthfulness in a given text.
To see what we mean, suppose two different language models answer a user’s question in two different ways, by generating two different answer texts. To compare these systems’ performance, we would need a “score card” that allowed us to objectively score the two texts as regards their factual correctness, using a variety of rubrics. Such a score card would allow us to record how often each type of error occurs in a given text, and aggregate the result into an overall truthfulness score for that text. Of particular importance would be the weighing of errors: large errors (e.g., a temperature reading that is very far from the actual temperature) should weigh more heavily than small ones, key facts should weigh more heavily than side issues, and errors that are genuinely misleading should weigh more heavily than typos that readers can correct by themselves. Essentially, the score card would work like a fair school teacher who marks pupils’ papers.
We have developed protocols for human evaluators to find factual errors in generated texts, as have other researchers, but we cannot yet create a score card as described above because we cannot assess the impact of individual errors.
What is needed, we believe, is a new strand of linguistically informed research, to tease out all the different parameters of “lying” in a manner that can inform the above-mentioned score cards, and that may one day be implemented into a reliable fact-checking protocol or algorithm. Until that time, those of us who are trying to assess the truthfulness of ChatGPT will be groping in the dark.
Featured image by Google DeepMind Via Unsplash (public domain)
Will artificial intelligence fatally undermine the integrity of scholarly publishing? A formal debate from the annual meeting of the Society for Scholarly Publishing.
The post SSP Conference Debate: AI and the Integrity of Scholarly Publishing appeared first on The Scholarly Kitchen.
Real patterns and the structure of language
There’s been a lot of hype recently about the emergence of technologies like ChatGPT and the effects they will have on science and society. Linguists have been especially curious about what highly successful large language models (LLMs) mean for their business. Are these models unearthing the hidden structure of language itself or just marking associations for predictive purposes?
In order to answer these sorts of questions we need to delve into the philosophy of what language is. For instance, if Language (with a big “L”) is an emergent human phenomenon arising from our communicative endeavours, i.e. a social entity, then AI is still some ways off approaching it in a meaningful way. If Chomsky, and those who follow his work, are correct that language is a modular mental system innately given to human infants and activated by miniscule amounts of external stimulus, then AI is again unlikely to be linguistic, since most of our most impressive LLMs are sucking up so many resources (both in terms of data and energy) that they are far from this childish learning target. On the third hand, if languages are just very large (possibly infinite) collections of sentences produced by applying discrete rules, then AI could be super-linguistic.
In my new book, I attempt to find a middle ground or intersection between these views. I start with an ontological picture (meaning a picture of what there is “out there”) advocated in the early nineties by the prominent philosopher and cognitive scientist, Daniel Dennett. He draws from information theory to distinguish between noise and patterns. In the noise, nothing is predictable, he says. But more often than not, we can and do find regularities in large data structures. These regularities provide us with the first steps towards pattern recognition. Another way to put this is that if you want to send a message and you need the entire series (string or bitmap) of information to do so, then it’s random. But if there’s some way to compress the information, it’s a pattern! What makes a pattern real, is whether or not it needs an observer for its existence. Dennett uses this view to make a case for “mild realism” about the mind and the position (which he calls the “intentional stance”) we use to identify minds in other humans, non-humans, and even artifacts. Basically, it’s like a theory we use to predict behaviour based on the success of our “minded” vocabulary comprising beliefs, desires, thoughts, etc. For Dennett, prediction matters theoretically!
If it’s not super clear yet, consider a barcode. At first blush, the black lines of varying length set to a background of white might seem random. But the lines (and spaces) can be set at regular intervals to reveal an underlying pattern that can be used to encode information (about the labelled entity/product). Barcodes are unique patterns, i.e. representations of the data from which more information can be drawn (by the way Nature produces these kinds of patterns too in fractal formation).
“The methodological chasm between theoretical and computational linguistics can be surmounted.”
I adapt this idea in two ways in light of recent advances in computational linguistics and AI. The first reinterprets grammars, specifically discrete grammars of theoretical linguistics, as compression algorithms. So, in essence, a language is like a real pattern. Our grammars are collections of rules that compress these patterns. In English, noticing that a sentence is made up of a noun phrase and verb phrase is such a compression. More complex rules capture more complex patterns. Secondly, discrete rules are just a subset of continuous processes. In other words, at one level information theory looks very statistical while generative grammar looks very categorical. But the latter is a special case of the former. I show in the book how some of the foundational theorems of information theory can be translated to discrete grammar representations. So there’s no need to banish the kinds of (stochastic) processes often used and manipulated in computational linguistics, as many theoretical linguists have been wont to do in the past.
This just means that the methodological chasm between theoretical and computational linguistics, which has often served to close the lines of communication between the fields, can be surmounted. Ontologically speaking, languages are not collections of sentences, minimal mental structures, or social entities by themselves. They are informational states taken from complex interactions of all of the above and more (like the environment). On this view, linguistics quickly emerges as a complexity science in which the tools of linguistic grammars, LLMs, and sociolinguistic observations all find a homogeneous home. Recent work on complex systems, especially in biological systems theory, has breathed new life into this interdisciplinary field of inquiry. I argue that the study of language, including the inner workings of both the human mind and ChatGPT, belong within this growing framework.
For decades, computational and theoretical linguists have been talking different languages. The shocking syntactic successes of modern LLMs and ChatGPT have forced them into the same room. Realising that languages are real patterns emerging from biological systems gets someone to break the awkward silence…
Featured image by Google DeepMind Via Unsplash (public domain)
Real patterns and the structure of language
There’s been a lot of hype recently about the emergence of technologies like ChatGPT and the effects they will have on science and society. Linguists have been especially curious about what highly successful large language models (LLMs) mean for their business. Are these models unearthing the hidden structure of language itself or just marking associations for predictive purposes?
In order to answer these sorts of questions we need to delve into the philosophy of what language is. For instance, if Language (with a big “L”) is an emergent human phenomenon arising from our communicative endeavours, i.e. a social entity, then AI is still some ways off approaching it in a meaningful way. If Chomsky, and those who follow his work, are correct that language is a modular mental system innately given to human infants and activated by miniscule amounts of external stimulus, then AI is again unlikely to be linguistic, since most of our most impressive LLMs are sucking up so many resources (both in terms of data and energy) that they are far from this childish learning target. On the third hand, if languages are just very large (possibly infinite) collections of sentences produced by applying discrete rules, then AI could be super-linguistic.
In my new book, I attempt to find a middle ground or intersection between these views. I start with an ontological picture (meaning a picture of what there is “out there”) advocated in the early nineties by the prominent philosopher and cognitive scientist, Daniel Dennett. He draws from information theory to distinguish between noise and patterns. In the noise, nothing is predictable, he says. But more often than not, we can and do find regularities in large data structures. These regularities provide us with the first steps towards pattern recognition. Another way to put this is that if you want to send a message and you need the entire series (string or bitmap) of information to do so, then it’s random. But if there’s some way to compress the information, it’s a pattern! What makes a pattern real, is whether or not it needs an observer for its existence. Dennett uses this view to make a case for “mild realism” about the mind and the position (which he calls the “intentional stance”) we use to identify minds in other humans, non-humans, and even artifacts. Basically, it’s like a theory we use to predict behaviour based on the success of our “minded” vocabulary comprising beliefs, desires, thoughts, etc. For Dennett, prediction matters theoretically!
If it’s not super clear yet, consider a barcode. At first blush, the black lines of varying length set to a background of white might seem random. But the lines (and spaces) can be set at regular intervals to reveal an underlying pattern that can be used to encode information (about the labelled entity/product). Barcodes are unique patterns, i.e. representations of the data from which more information can be drawn (by the way Nature produces these kinds of patterns too in fractal formation).
“The methodological chasm between theoretical and computational linguistics can be surmounted.”
I adapt this idea in two ways in light of recent advances in computational linguistics and AI. The first reinterprets grammars, specifically discrete grammars of theoretical linguistics, as compression algorithms. So, in essence, a language is like a real pattern. Our grammars are collections of rules that compress these patterns. In English, noticing that a sentence is made up of a noun phrase and verb phrase is such a compression. More complex rules capture more complex patterns. Secondly, discrete rules are just a subset of continuous processes. In other words, at one level information theory looks very statistical while generative grammar looks very categorical. But the latter is a special case of the former. I show in the book how some of the foundational theorems of information theory can be translated to discrete grammar representations. So there’s no need to banish the kinds of (stochastic) processes often used and manipulated in computational linguistics, as many theoretical linguists have been wont to do in the past.
This just means that the methodological chasm between theoretical and computational linguistics, which has often served to close the lines of communication between the fields, can be surmounted. Ontologically speaking, languages are not collections of sentences, minimal mental structures, or social entities by themselves. They are informational states taken from complex interactions of all of the above and more (like the environment). On this view, linguistics quickly emerges as a complexity science in which the tools of linguistic grammars, LLMs, and sociolinguistic observations all find a homogeneous home. Recent work on complex systems, especially in biological systems theory, has breathed new life into this interdisciplinary field of inquiry. I argue that the study of language, including the inner workings of both the human mind and ChatGPT, belong within this growing framework.
For decades, computational and theoretical linguists have been talking different languages. The shocking syntactic successes of modern LLMs and ChatGPT have forced them into the same room. Realising that languages are real patterns emerging from biological systems gets someone to break the awkward silence…
Featured image by Google DeepMind Via Unsplash (public domain)
What can Large Language Models offer to linguists?
It is fair to say that the field of linguistics is hardly ever in the news. That is not the case for language itself and all things to do with language—from word of the year announcements to countless discussions about grammar peeves, correct spelling, or writing style. This has changed somewhat recently with the proliferation of Large Language Models (LLMs), and in particular since the release of OpenAI’s ChatGPT, the best-known language model. But does the recent, impressive performance of LLMs have any repercussions for the way in which linguists carry out their work? And what is a Language Model anyway?
At heart, all an LLM does is predict the next word given a string of words as a context —that is, it predicts the next, most likely word. This is of course not what a user experiences when dealing with language models such as ChatGPT. This is on account of the fact that ChatGPT is more properly described as a “dialogue management system”, an AI “assistant” or chatbot that translates a user’s questions (or “prompts”) into inputs that the underlying LLM can understand (the latest version of OpenAI’s LLM is a fine-tuned version of GPT-4).
“At heart, all an LLM does is predict the next word given a string of words as a context.”
An LLM, after all, is nothing more than a mathematical model in terms of a neural network with input layers, output layers, and many deep layers in between, plus a set of trained “parameters.” As the computer scientist Murray Shanahan has put it in a recent paper, when one asks a chatbot such as ChatGPT who was the first person to walk on the moon, what the LLM is fed is something along the lines of:
Given the statistical distribution of words in the vast public corpus of (English) text, what word is most likely to follow the sequence “The first person to walk on the Moon was”?
That is, given an input such as the first person to walk on the Moon was, the LLM returns the most likely word to follow this string. How have LLMs learned to do this? As mentioned, LLMs calculate the probability of the next word given a string of words, and it does so by representing these words as vectors of values from which to calculate the probability of each word, and where sentences can also be represented as vectors of values. Since 2017, most LLMs have been using “transformers,” which allow the models to carry out matrix calculations over these vectors, and the more transformers are employed, the more accurate the predictions are—GPT-3 has some 96 layers of such transformers.
The illusion that one is having a conversation with a rational agent, for it is an illusion, after all, is the result of embedding an LLM in a larger computer system that includes background “prefixes” to coax the system into producing behaviour that feels like a conversation (the prefixes include templates of what a conversation looks like). But what the LLM itself does is generate sequences of words that are statistically likely to follow from a specific prompt.
It is through the use of prompt prefixes that LLMs can be coaxed into “performing” various tasks beyond dialoguing, such as reasoning or, according to some linguists and cognitive scientists, learn the hierarchical structures of a language (this literature is ever increasing). But the model itself remains a sequence predictor, as it does not manipulate the typical structured representations of a language directly, and it has no understanding of what a word or a sentence means—and meaning is a crucial property of language.
An LLM seems to produce sentences and text like a human does—it seems to have mastered the rules of the grammar of English—but at the same time it produces sentences based on probabilities rather on the meanings and thoughts to express, which is how a human person produces language. So, what is language so that an LLM could learn it?
“An LLM seems to produce sentences like a human does but it produces them based on probabilities rather than on meaning.”
A typical characterisation of language is as a system of communication (or, for some linguists, as a system for having thoughts), and such a system would include a vocabulary (the words of a language) and a grammar. By a “grammar,” most linguists have in mind various components, at the very least syntax, semantics, and phonetics/phonology. In fact, a classic way to describe a language in linguistics is as a system that connects sound (or in terms of other ways to produce language, such as hand gestures or signs) and meaning, the connection between sound and meaning mediated by syntax. As such, every sentence of a language is the result of all these components—phonology, semantics, and syntax—aligning with each other appropriately, and I do not know of any linguistic theory for which this is not true, regardless of differences in focus or else.
What this means for the question of what LLMs can offer linguistics, and linguists, revolves around the issue of what exactly LLMs have learned to begin with. They haven’t, as a matter of fact, learned a natural language at all, for they know nothing about phonology or meaning; what they have learned is the statistical distribution of the words of the large texts they have been fed during training, and this is a rather different matter.
As has been the case in the past with other approaches in computational linguistics and natural language processing, LLMs will certainly flourish within these subdisciplines of linguistics, but the daily work of a regular linguist is not going to change much any time soon. Some linguists do study the properties of texts, but this is not the most common undertaking in linguistics. Having said that, how about the opposite question: does a run-of-the-mill linguist have much to offer to LLMs and chatbots at all?
Featured image: Google Deepmind via Unsplash (public domain)
Download the transcript of this interview.
For this episode of Platypod, I talked to Dr. Tanja Ahlin about her research, work, and academic trajectory. She’s currently a postdoctoral researcher at the University of Amsterdam in the Netherlands, and her work focuses on intersections of medical anthropology, social robots, and artificial intelligence. I told her of my perspective as a grad student, making plans and deciding what routes to take to be successful in my field. Dr. Ahlin was very generous in sharing her stories and experiences, which I’m sure are helpful to other grad students as well. Enjoy this episode, and contact us if you have questions, thoughts, or suggestions for other episodes.
Dr. Tanja Ahlin, image from her personal website.
Dr. Tanja Ahlin is a medical anthropologist and STS scholar with a background in translation. She has translated books about technology and more. She has a master’s degree in medical anthropology, focusing on the topic of health and society in South Asia. Dr. Ahlin has been interested in e-health/telehealth for a long time, before the recent COVID-19 pandemic years, in which those words became part of our daily vocabulary. Her Ph.D., which she concluded at the University of Amsterdam, has focused on everyday digital technologies in elder care at a distance. Her Ph.D. research is being published as a book at Rutgers University Press. The book will be available for purchase starting on August 11, 2023.
Calling Family – Digital Technologies and the Making of Transnational Care Collectives | Rutgers University Press
In our conversation, we talked about Dr. Ahlin’s blog focusing on the Anthropology of Data and AI. This project—in which Dr. Ahlin writes about the intersection of tech and different fields such as robotics, policy, ethics, health, and ethnography—is a kind of translation work, since Dr. Ahlin is writing about complex topics to a broader audience who are not familiar with some STS and anthropological concepts and discussions. “The blog posts are not supposed to be very long. I aim for two to four minutes of reading … I realized that people often don’t have time to read more than that, right?” says Dr. Tanja Ahlin.
Dr. Ahlin’s book is based on ten years of ethnographic research with Indian transnational families. These are families where family members live all around the world. The reason for migration is mostly due to work opportunities abroad. In her research, Dr. Ahlin looked at how these families used all kinds of technologies like mobile phones and webcams, the Internet, and Whatsapp, not only to keep in touch with each other but also to provide care at a distance. Dr. Ahlin conducted interviews with nurses living all around the world, from the US to Canada to the UK, the Maldives, and Australia. This varied and diverse field gave origin to the concept of field events that Dr. Ahlin develops in her work. In her work, Dr. Ahlin also developed the notion of transnational care collective to show how care is reconceptualized when it has to be done at a distance.
In sum, this episode of Platypod highlights how anthropologists come from different backgrounds and gives an honest overview of how we get to research our topics and occupy the spaces we do. We do not have linear stories, and that does not determine our potential. We at Platypod are very thankful for Dr. Ahlin’s time and generosity.
During the summer slow-down, many news items will be consolidated in occasional “philosophy news” summary posts. This is the first.
Discussion welcome.
The post Philosophy News Summary first appeared on Daily Nous.
Our work at Lumen is focused on eliminating race and income as predictors of student success in the US postsecondary setting. One thing we’ve learned as we’ve worked to erase this persistent gap in academic performance is that it is far easier to “slide the gap to the right” than it is to close it. In other words, interventions intended to benefit the lowest performing students often benefit all students, so that everyone’s academic performance improves. That’s great from one perspective – everyone learned more! But rather than decreasing the size of the gap, these interventions leave the gap in tact and nudge it up the scale to the right. Interventions that have an accurately targeted effect can be hard to find.
For this reason, I was particularly excited to see Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, a new study from researchers at MIT which finds that access to ChatGPT dramatically reduces productivity inequality on writing tasks. The abstract reads:
We examine the productivity effects of a generative artificial intelligence technology—the assistive chatbot ChatGPT—in the context of mid-level professional writing tasks. In a preregistered online experiment, we assign occupation-specific, incentivized writing tasks to 444 college-educated professionals, and randomly expose half of them to ChatGPT. Our results show that ChatGPT substantially raises average productivity: time taken decreases by 0.8 SDs [37%, or from 27 minutes to 17 minutes] and output quality rises by 0.4 SDs [a 0.75 point increase in grade on a 7 point scale]. Inequality between workers decreases, as ChatGPT compresses the productivity distribution by benefiting low-ability workers more. ChatGPT mostly substitutes for worker effort rather than complementing worker skills, and restructures tasks towards idea-generation and editing and away from rough-drafting. Exposure to ChatGPT increases job satisfaction and self-efficacy and heightens both concern and excitement about automation technologies.
That kind of result gets me excited! Simultaneously decreasing inequity while increasing self-efficacy and satisfaction? Yes, please!
Section 2.3 explicitly discusses productivity inequality, describing how access to ChatGPT helped close that gap:
The control group exhibits persistent productivity inequality: participants who score well on the first task also tend to score well on the second task. As Figure 2 Panel (a) shows, there is a correlation of 0.49 between a control participant’s average grade on the first task and their average grade on the second task. In the treatment group, initial inequalities are half-erased by the treatment [access to ChatGPT]: the correlation between first-task and second-task grades is only 0.25 (p-value on difference in slopes = 0.004). This reduction in inequality is driven by the fact that participants who scored lower on the first round benefit more from ChatGPT access, as the figure shows: the gap between the treatment and control lines is much larger at the left-hand end of the x-axis. (p. 5)
There seems to be real promise here for making progress toward closing the equity gap in education. However, what we see positively as “productivity gains” in the world of work is often seen negatively as “cheating” in the world of school. And while there are certainly challenges to navigate here, results like those in this paper from MIT make our efforts to navigate them effectively all the more critical as we work to close the equity gap.
Well Colombia have agreed to publish a second dark, dark, intense book. Clearly they couldn't get enough of Dark Ecology, the Wellek Lectures that I gave in the lineage of Cixous and Balibar, and now they're going to work with me to get this one out for Earth Day 2024.
It's a book about slavery, racism, capitalism, AI, ecology, despair, religion and mysticism. It's freaking AWESOME. I wasn't quite ready to say stuff like that out loud when I last worked for Columbia.
Because Columbia have done this, I'm now committing to them. I've been living my life waiting for approval and love in so many ways and I am DONE. Love is a thing you DO not a thing you wait for. That phenomenology cashes out to being a theater critic and in the end, all the plays are bad. Because you're waiting for them to be bad. I got really good at getting up and leaving the theater no matter where the play was at: I could see the writing on the wall. So at least there's that. But that is still...that.
So this is my proclamation to the world. I'm with Columbia now. Like how my best buddy Jeff Kripal (the X Men actors have to read his work when they're on set) is with Chicago. It's a great thing. I kept thinking when I found that out, he must have a lot of space in his soul to think, he doesn't have to keep waiting for people to say yes like me, who acts like they're a character in a Jane Austen novel.
That never occurred to me until this week, when Columbia accepted Hell. But it had occurred to me in my personal life, in part because my mum's family traces their lineage back to the lower gentry in the later eighteenth century. And that's a horrible precarious place to be. My grandmother to cap it all was Welsh lower gentry. Imagine Sense and Sensibility, but set in Wales. Just horrible ancient colonial vibes. You're dead unless Mister Right sweeps you off your feet. So you have to sit around ever so politely waiting for Mister Right, not putting a foot wrong, including doing a single day of work, and you can't access your own money until said Mister Right shows up.
This was me and book contracts. I thought it was great, a kind of naive drifting that meant I wasn't pushy and manipulative, and I'm not. But this is better. I'm not Elinor Dashwood. That energy crippled my family. Austen novels are about the terrible pain of a precarious class, women in the lower gentry during a time of enclosure and transition from primitive accumulation to automated capitalism.
Hell is about masters and slaves. Hell is about the Devil. Hell is about the biosphere as the Devil and ideas about the Devil as the Devil that's burning the biosphere.
Hell is also about AI. ery directly, because it’s totally relevant. I think the real driver here is the master slave template that drives everything else (subject versus object, male versus female, active versus passive…). We need to abolish that template. The idea of creating the perfect slave that is then the perfect master is basically every story about selling one’s soul to Satan.
Treating the biosphere like that, because treating each other like that, is why AI people are blundering into this and why that feeling of “the search for AI is like an unstoppable AI” keeps happening…
Digital dilemmas: feminism, ethics, and the cultural implications of AI [podcast]
Skynet. HAL 9000. Ultron. The Matrix. Fictional depictions of artificial intelligences have played a major role in Western pop culture for decades. While nowhere near that nefarious or powerful, real AI has been making incredible strides and, in 2023, has been a big topic of conversation in the news with the rapid development of new technologies, the use of AI generated images, and AI chatbots such as ChatGPT becoming freely accessible to the general public.
On today’s episode, we welcomed Dr Kerry McInerney and Dr Eleanor Drage, editors of Feminist AI: Critical Perspectives on Data, Algorithms and Intelligent Machines, and then Dr Kanta Dihal, co-editor of Imagining AI: How the World Sees Intelligent Machines, to discuss how AI can be influenced by culture, feminism, and Western narratives defined by popular TV shows and films. Should AI be accessible to all? How does gender influence the way AI is made? And most importantly, what are the hopes and fears for the future of AI?
Check out Episode 82 of The Oxford Comment and subscribe to The Oxford Comment podcast through your favourite podcast app to listen to the latest insights from our expert authors.
Look out for Feminist AI: Critical Perspectives on Algorithms, Data, and Intelligent Machines, edited by Jude Browne, Stephen Cave, Eleanor Drage, and Kerry McInerney, which publishes in the UK in August 2023 and in the US in October 2023.
If you want to hear more from Dr Eleanor Drage and Dr Kerry McInerney, you can listen to their podcast: The Good Robot Podcast on Gender, Feminism and Technology.
In May 2023, the Open Access title, Imagining AI: How the World Sees Intelligent Machines, edited by Stephen Cave and Kanta Dihal publishes in the UK; it publishes in the US in July 2023.
You may also be interested in AI Narratives: A History of Imaginative Thinking about Intelligent Machines, edited by Stephen Cave, Kanta Dihal, and Sarah Dillon, which looks both at classic AI to the modern age, and contemporary narratives.
You can read the following two chapters from AI Narratives for free until 31 May:
Other relevant book titles include:
You may also be interested in the following journal articles:
Featured image: ChatGPT homepage by Jonathan Kemper, CC0 via Unsplash.
If you can remember the web of 30 years ago(!), you can remember a time when all it took to make a website was a little knowledge of HTML and a tilde account on the university VAXcluster (e.g., /~wiley6/). While it’s still possible to make a simple website today with just HTML, making modern websites requires a dizzying array of technical skills, including HTML, CSS, JavaScript frameworks, databases and SQL, cloud devops, and others. While these websites require far more technical expertise to build, they are also far more feature-rich and functional then their ancestors of 30 years ago. (Imagine trying to code each of the millions of pages on Wikipedia.org or Amazon.com completely by hand with notepad!)
This is what large language models (LLMs) like ChatGPT are doing to OER. Next generation OER will not be open textbooks that were created faster or more efficiently because LLMs wrote first drafts in minutes. That’s current generation OER simply made more efficiently. The next generation of OER will be the embeddings (from a 5R perspective, these are revised versions of an OER) that are part of the process of feeding domain knowledge into LLMs so that they can answer questions correctly and give you accurate explanations and examples. Creating embeddings and injecting this additional context into an LLM just-in-time as part of a prompt engineering strategy requires significantly more technical skill than typing words into Pressbooks does. But it will also give us OER that are far more feature-rich and functional than their open ancestors of 25 years ago.
Here’s a video tutorial of how to integrate a specific set of domain knowledge into GPT3 so that it can dialog with a user based on that specific domain knowledge. This domain knowledge could come from chapters in an open textbook, but in the example in the video it’s coming from software documentation. Granted, this video is almost two months old, which feels more than two years old at the rate AI is changing right now. So this isn’t the exact way we’ll end up doing it, but the video will give you the idea.
Rather than fine tuning an LLM, where the entire model training process has to be repeated, embeddings allow us to find just the right little pieces of OER to provide to the LLM as additional context when we submit a prompt. This is orders of magnitude faster and less expensive than retraining the entire model, and still gives the model access to the domain specific information we want it to have during our conversation / tutoring session / etc. And by “orders of magnitude faster and less expensive” I mean this is a legitimate option for a normal person with some technical skill, unlike retraining a model which can easily cost over $1M in compute alone.
Every day feels like a year for those of us trying to keep up with what’s happening with AI right now. It would be the understatement of the century to say lots more will happen in this space – we’re literally just scratching the surface. Our collective lack of imagination is the only thing holding us back. What an incredible time to be a learner! What an incredible time to be a teacher! What an incredible time to be working and researching in edtech!
“We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.”
As of Tuesday night, over 1100 people had signed the letter, including philosophers such as Seth Lazar (of the Machine Intelligence and Normative Theory Lab at ANU), James Maclaurin (Co-director Centre for AI and Public Policy at Otago University), and Huw Price (Cambridge, former Director of the Leverhulme Centre for the Future of Intelligence), scientists such as Yoshua Bengio (Director of the Mila – Quebec AI Institute at the University of Montreal), Victoria Krakovna (DeepMind, co-founder of Future of Life Institute), Stuart Russell (Director of the Center for Intelligent Systems at Berkeley), and Max Tegmark (MIT Center for Artificial Intelligence & Fundamental Interactions), and tech entrepreneurs such as Elon Musk (SpaceX, Tesla, Twitter), Jaan Tallinn (Co-Founder of Skype, Co-Founder of the Centre for the Study of Existential Risk at Cambridge), and Steve Wozniak (co-founder of Apple), and many others.
Pointing out some of the risks of AI, the letter decries the “out-of-control race to develop and deploy ever more powerful digital minds that no one—not even their creators—can understand, predict, or reliably control” and the lack of “planning and management” appropriate to the potentially highly disruptive technology.
Here’s the full text of the letter (references omitted):
AI systems with human-competitive intelligence can pose profound risks to society and humanity, as shown by extensive research and acknowledged by top AI labs. As stated in the widely-endorsed Asilomar AI Principles, Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources. Unfortunately, this level of planning and management is not happening, even though recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one—not even their creators—can understand, predict, or reliably control.
Contemporary AI systems are now becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable. This confidence must be well justified and increase with the magnitude of a system’s potential effects. OpenAI’s recent statement regarding artificial general intelligence, states that “At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models.” We agree. That point is now.
Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.
AI labs and independent experts should use this pause to jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts. These protocols should ensure that systems adhering to them are safe beyond a reasonable doubt. This does not mean a pause on AI development in general, merely a stepping back from the dangerous race to ever-larger unpredictable black-box models with emergent capabilities.
AI research and development should be refocused on making today’s powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal.
In parallel, AI developers must work with policymakers to dramatically accelerate development of robust AI governance systems. These should at a minimum include: new and capable regulatory authorities dedicated to AI; oversight and tracking of highly capable AI systems and large pools of computational capability; provenance and watermarking systems to help distinguish real from synthetic and to track model leaks; a robust auditing and certification ecosystem; liability for AI-caused harm; robust public funding for technical AI safety research; and well-resourced institutions for coping with the dramatic economic and political disruptions (especially to democracy) that AI will cause.
Humanity can enjoy a flourishing future with AI. Having succeeded in creating powerful AI systems, we can now enjoy an “AI summer” in which we reap the rewards, engineer these systems for the clear benefit of all, and give society a chance to adapt. Society has hit pause on other technologies with potentially catastrophic effects on society. We can do so here. Let’s enjoy a long AI summer, not rush unprepared into a fall.
The letter is here. It is published by the Future of Life Institute, which supports “the development of institutions and visions necessary to manage world-driving technologies and enable a positive future” and aims to “reduce large-scale harm, catastrophe, and existential risk resulting from accidental or intentional misuse of transformative technologies.”
Discussion welcome.
Related: “Thinking About Life with AI“, “Philosophers on Next-Generation Large Language Models“, “GPT-4 and the Question of Intelligence“, “We’re Not Ready for the AI on the Horizon, But People Are Trying”
“What kind of civilization is it that turns away from the challenge of dealing with more… intelligence?”
That’s Tyler Cowen (GMU), writing at Marginal Revolution. He is addressing the “radical uncertainty” we should acknowledge regarding a future in which we’ve developed artificial intelligence (AI). Even if one does not believe that large language models (LLMs) could be a form of AI (recall the possible architectural limitation noted in the paper discussed last week), it does seem that at least the AI-like is here, will only get more convincing in functionality, and will likely bring substantial changes to our lives.
Cowen’s targets are those who are making broad judgments about the goodness and badness of these technological developments. He thinks we’re living in a transformational period—he calls it “moving history”—and our predictions about it should be informed by an appropriate degree of epistemic humility. He says:
Since we are not used to living in moving history, and indeed most of us are psychologically unable to truly imagine living in moving history, all these new AI developments pose a great conundrum. We don’t know how to respond psychologically, or for that matter substantively. And just about all of the responses I am seeing I interpret as “copes,” whether from the optimists, the pessimists, or the extreme pessimists… No matter how positive or negative the overall calculus of cost and benefit, AI is very likely to overturn most of our apple carts, most of all for the so-called chattering classes.
Of course, that AI is “very likely to overturn most of our apple carts” and will ultimately be as unpredictable in its effects as the invention of fire or the printing press is itself a bold prediction. But suppose we accept it. That we can’t be certain of what might happen doesn’t render speculation random or pointless.
So let’s speculate. I’m curious what changes, if any, you think we might be in for.
And let’s talk about how to speculate. I’m curious about how to think about these changes.
We might learn something from paleo-futurology, the study of past predictions of the future. One lesson appears to be that while some technological advances may be easy to predict, social changes are less so. Futurists of the 1950s, thinking about life in the year 2000, were able to anticipate, in some form, for example, video calls, increased use of plastics, and easier-to-clean fabrics:
Some of the pictures that accompanied “Miracles You’ll See in the Next Fifty Years” by Waldemar Kaempffert, published in Popular Mechanics in February, 1950
Yet apparently it was not as easy to predict how odd it would be to relegate the shopping and cleaning to “the housewife of 2000”.
Technological changes affect attitudes and norms that in turn affect our expectations for various aspects of our lives, and those expectations have effects on how we live, what we think, the kinds of individual and collective problems we recognize, what else we are spurred to change, and so on.
So it is complicated, and so yes, let’s be epistemically humble. But let’s let our imaginations roam a bit, too, to explore the possibilities.
Written by David Lyreskog
‘Hivemind Brain-Computer Interfaces’, as imagined by the AI art generator Midjourney
A growing number of technologies are currently being developed to improve and distribute thinking and decision-making. Rapid progress in brain-to-brain interfacing, and hybrid and artificial intelligence, promises to transform how we think about collective and collaborative cognitive tasks. With implementations ranging from research to entertainment, and from therapeutics to military applications, as these tools continue to improve, we need to anticipate and monitor their impacts – how they may affect our society, but also how they may reshape our fundamental understanding of agency, responsibility, and other concepts which ground our moral landscapes.
In a new paper, I, together with Dr. Hazem Zohny, Prof. Julian Savulescu, and Prof. Ilina Singh, show how these new technologies may reshape fundamental components of widely accepted concepts pertaining to moral behaviour. The paper, titled ‘Merging Minds: The Conceptual and Ethical Impacts of Emerging Technologies for Collective Minds’, was just published in Neuroethics, and is freely available as an Open Access article through the link above.
In the paper, we argue that the received views on how we (should) ascribe responsibility to individuals and collectives map poorly onto networks of these ‘Collective Minds’. The intimately collective nature of direct multiple-brain interfaces, for instance, where human minds can collaborate on and complete complex tasks without necessarily being in the same room – or even on the same continent! – seem to suggest a collectivist moral framework to ascribe agency and responsibility. However, the technologies we are seeing in R&D do not necessitate the meeting of criteria we normally would turn to for ascription of such frameworks; they do not, for instance, seem to rely on that participants have shared goals, know what the goals of other participants are, or even know whether they are collaborating with another person or a computer.
In anticipating and assessing the ethical impacts of Collective Minds, we propose that we move beyond binary approaches to thinking about agency and responsibility (i.e. that they are either individual or collective), and that relevant frameworks for now focus on other aspects of significance to ethical analysis, such as (a) technical specifications of the Collective Mind, (b) the domain in which the technology is deployed, and (c) the reversibility of its physical and mental impacts. However, in the future, we will arguably need to find other ways to assess agency constellations and responsibility distribution, lest we abandon these concepts completely in this domain.
FreshRSS 