Login
Our work at Lumen is focused on eliminating race and income as predictors of student success in the US postsecondary setting. One thing we’ve learned as we’ve worked to erase this persistent gap in academic performance is that it is far easier to “slide the gap to the right” than it is to close it. In other words, interventions intended to benefit the lowest performing students often benefit all students, so that everyone’s academic performance improves. That’s great from one perspective – everyone learned more! But rather than decreasing the size of the gap, these interventions leave the gap in tact and nudge it up the scale to the right. Interventions that have an accurately targeted effect can be hard to find.
For this reason, I was particularly excited to see Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, a new study from researchers at MIT which finds that access to ChatGPT dramatically reduces productivity inequality on writing tasks. The abstract reads:
We examine the productivity effects of a generative artificial intelligence technology—the assistive chatbot ChatGPT—in the context of mid-level professional writing tasks. In a preregistered online experiment, we assign occupation-specific, incentivized writing tasks to 444 college-educated professionals, and randomly expose half of them to ChatGPT. Our results show that ChatGPT substantially raises average productivity: time taken decreases by 0.8 SDs [37%, or from 27 minutes to 17 minutes] and output quality rises by 0.4 SDs [a 0.75 point increase in grade on a 7 point scale]. Inequality between workers decreases, as ChatGPT compresses the productivity distribution by benefiting low-ability workers more. ChatGPT mostly substitutes for worker effort rather than complementing worker skills, and restructures tasks towards idea-generation and editing and away from rough-drafting. Exposure to ChatGPT increases job satisfaction and self-efficacy and heightens both concern and excitement about automation technologies.
That kind of result gets me excited! Simultaneously decreasing inequity while increasing self-efficacy and satisfaction? Yes, please!
Section 2.3 explicitly discusses productivity inequality, describing how access to ChatGPT helped close that gap:
The control group exhibits persistent productivity inequality: participants who score well on the first task also tend to score well on the second task. As Figure 2 Panel (a) shows, there is a correlation of 0.49 between a control participant’s average grade on the first task and their average grade on the second task. In the treatment group, initial inequalities are half-erased by the treatment [access to ChatGPT]: the correlation between first-task and second-task grades is only 0.25 (p-value on difference in slopes = 0.004). This reduction in inequality is driven by the fact that participants who scored lower on the first round benefit more from ChatGPT access, as the figure shows: the gap between the treatment and control lines is much larger at the left-hand end of the x-axis. (p. 5)
There seems to be real promise here for making progress toward closing the equity gap in education. However, what we see positively as “productivity gains” in the world of work is often seen negatively as “cheating” in the world of school. And while there are certainly challenges to navigate here, results like those in this paper from MIT make our efforts to navigate them effectively all the more critical as we work to close the equity gap.
If you can remember the web of 30 years ago(!), you can remember a time when all it took to make a website was a little knowledge of HTML and a tilde account on the university VAXcluster (e.g., /~wiley6/). While it’s still possible to make a simple website today with just HTML, making modern websites requires a dizzying array of technical skills, including HTML, CSS, JavaScript frameworks, databases and SQL, cloud devops, and others. While these websites require far more technical expertise to build, they are also far more feature-rich and functional then their ancestors of 30 years ago. (Imagine trying to code each of the millions of pages on Wikipedia.org or Amazon.com completely by hand with notepad!)
This is what large language models (LLMs) like ChatGPT are doing to OER. Next generation OER will not be open textbooks that were created faster or more efficiently because LLMs wrote first drafts in minutes. That’s current generation OER simply made more efficiently. The next generation of OER will be the embeddings (from a 5R perspective, these are revised versions of an OER) that are part of the process of feeding domain knowledge into LLMs so that they can answer questions correctly and give you accurate explanations and examples. Creating embeddings and injecting this additional context into an LLM just-in-time as part of a prompt engineering strategy requires significantly more technical skill than typing words into Pressbooks does. But it will also give us OER that are far more feature-rich and functional than their open ancestors of 25 years ago.
Here’s a video tutorial of how to integrate a specific set of domain knowledge into GPT3 so that it can dialog with a user based on that specific domain knowledge. This domain knowledge could come from chapters in an open textbook, but in the example in the video it’s coming from software documentation. Granted, this video is almost two months old, which feels more than two years old at the rate AI is changing right now. So this isn’t the exact way we’ll end up doing it, but the video will give you the idea.
Rather than fine tuning an LLM, where the entire model training process has to be repeated, embeddings allow us to find just the right little pieces of OER to provide to the LLM as additional context when we submit a prompt. This is orders of magnitude faster and less expensive than retraining the entire model, and still gives the model access to the domain specific information we want it to have during our conversation / tutoring session / etc. And by “orders of magnitude faster and less expensive” I mean this is a legitimate option for a normal person with some technical skill, unlike retraining a model which can easily cost over $1M in compute alone.
Every day feels like a year for those of us trying to keep up with what’s happening with AI right now. It would be the understatement of the century to say lots more will happen in this space – we’re literally just scratching the surface. Our collective lack of imagination is the only thing holding us back. What an incredible time to be a learner! What an incredible time to be a teacher! What an incredible time to be working and researching in edtech!
At some point over the last decade, open educational resources (OER) advocacy in US higher education became zero textbook costs (ZTC) advocacy. The two are intertwined now in a manner that would be difficult to disentangle even if you wanted to try. There are plenty of practical reasons why this might have happened. For example, politicians understand costs much better than they understand learning, which makes policy work and other political advocacy around eliminating textbook costs far easier than advocating for ways that “open” (whatever that word means) might be leveraged to improve student outcomes. But OER / ZTC advocates have had a fundamental problem simmering for many years now, and the recent advent of large language models (LLMs) like GPT-4 will quickly bring that simmer to a boil.
The idea of “zero textbook costs” makes a kind of sense when you believe that the ideal instructional materials are books. Books contain static, unchanging words and pictures on a page. They’re perfect for printing or exporting as a PDF. And when you’re talking about a PDF – a digital file of which you can make seemingly infinite, perfect copies for fractions of a penny – I understand why one might expect the price of these files to eventually approach their true cost to deliver – free.
The elephant in the room is that the interactive capabilities of computers and the internet make it possible to create dramatically more effective educational materials than static books (whether print or PDF). The ability to provide learners with practice and immediate, corrective feedback is very powerful. CMU researchers have demonstrated that the learning effect of this kind of online interactive practice is approximately six times larger than the learning effects of either reading or watching video. However, these kinds of interactive learning materials have a cost associated with delivering them, which means that whether you get them CMU, OpenStax, Lumen Learning, or elsewhere, they typically have a price that is greater than zero. Consequently, because there is a price associated with using “courseware” (the term many people use for this kind of interactive educational resource), OER / ZTC supporters end up actively advocating against courseware and for static educational materials that will result in inferior learning outcomes for students – all in the name of saving them money. This is why I refer to this line of advocacy as “free no matter the cost.”
OER / ZTC advocates have largely succeeded in turning a blind eye to the courseware elephant in the room. But conversational LLMs like ChatGPT are a towering pyramid of elephants in the room. They take us all the way back to Bloom’s two sigma problem, in which Bloom and colleagues demonstrated that the average student – the average student – who is tutored full-time outperforms 98% of students who learn in a traditional classroom setting. Tutoring is an incredibly powerful teaching method, and LLMs have finally made this capability broadly available at a reasonable cost. There may be ineffective initial attempts, but eventually combining courseware with LLM-powered tutoring will dramatically improve outcomes for students who get the chance to use these tools.
But alas, some students won’t get that opportunity. As with courseware, LLMs have real costs associated with their hosting and delivery, meaning that learning materials that incorporate LLMs will have a price greater than zero. And this means that students whose faculty, department, institution, or state have guaranteed that students will pay zero dollars for their course materials will miss out on these next-generation learning materials.
Conversational LLMs like ChatGPT will confront OER / ZTC advocates with an even starker choice than courseware has in the past. Will they continue to advocate for “free no matter the cost,” even as it becomes more obvious that they are essentially advocating for decreased student learning? Maybe student complaints about OER / ZTC policies that, paradoxically, deny them access to these kinds of highly effective learning materials will finally move the learning materials discussion beyond “zero cost” and back to being focused on how we best support student learning. We can hope.
Is there a role for OER in this emerging learning materials landscape? I believe there will be. But that’s a topic for another time.
The journal Nature, and all other Springer Nature titles, have updated their Guide to Authors with rules for using LLMs like ChatGPT when writing research articles for the publication. To summarize, the rules say:
1. DO NOT list the LLM as an author, and
2. DO describe how you used the LLM in a Methods, Acknowledgments, or other appropriate section.
With a journal as prestigious as Nature having established formal guidelines, I expect other journals will adopt similar rules relatively quickly.
It’s interesting to see how different fields are grappling with the different issues raised by LLMs. As I wrote in my most recent post, the US Copyright Office has stated that works created by generative AI are not eligible for copyright protection because copyright is reserved for “‘the fruits of intellectual labor’ that ‘are founded in the creative powers of the [human] mind.'” In refusing to list an LLM as an author or co-author on a research article, Nature takes a different approach, explaining, “any attribution of authorship carries with it accountability for the work, and AI tools cannot take such responsibility.”
2022 saw some significant advancements in artificial intelligence. My threshold for “significant” here being that the advances moved out of labs and arXiv.org preprints and into tools that many people were using and talking about. Lots of people thought text-to-image tools like DALL-E, Stable Diffusion, and Midjourney were fun. But Large Language Models (LLMs), and particularly the recent demo of ChatGPT, seem to have put the fear of God into everyone from middle school English teachers to the CEO of Google. The potential partnership between OpenAI (the makers of ChatGPT) and Microsoft may even present the first substantive challenge to Google’s search monopoly we’ve ever seen – and that’s saying something. While most of the dialog around AI and education seems to be focused on assessment, I think the implications for instructional designers are critically important, too. And, because you’ve got to play the hits, let’s look at what their impact will be on OER as well.
I think a lot of people believe that instructional designers are the people who are really good at using the campus LMS – folks who can create new course shells for faculty, help faculty get their content uploaded into the LMS, and maybe even help with a little multimedia editing and production. Unfortunately, literally none of those things are instructional design. Instructional design is the process of leveraging what we understand about how people learn to create experiences that maximize the likelihood that the people who participate in those experiences will learn. Instructional designers need a deep understanding of both learning science research and educational technologies in order to effectively integrate them in support of learning. Uploading a syllabus PDF into Blackboard is not instructional design.
You know what else isn’t instructional design? The creation of accurate descriptions and explanations of facts, theories, and models. The raw text and images that make up the overwhelming majority of what we call textbooks. I’ve written previously about the difference between informational resources and educational resources. Wikipedia and other encyclopedias are informational resources. Reference materials and technical documentation are informational resources. The overwhelming majority of textbooks are primarily informational resources. What distinguishes an educational resource from an informational resource is that the latter shows some intentional application of knowledge about how people learn. I have previously argued that the minimum amount of effort you could invest to convert an informational resource into an educational resource was to add practice with feedback. That simple sounding task quickly explodes in complexity as you consider the research on what form that practice should take, how long it should last, when it should be scheduled, what kind of feedback should be provided, whether the feedback should come immediately or after some delay, etc.
Now, please don’t misunderstand me – there is absolutely an art and science to creating accurate, beautiful informational resources. It’s just a different art and science from the one involved in creating educational resources. All educational resources begin as informational resources, so they are critically important to the work we do as instructional designers. You might say they are a kind of intellectual infrastructure, in the way Brett Frischmann uses the term:
Intellectual infrastructure, such as basic research, ideas, general purpose technologies, and languages, creates benefits for society primarily by facilitating a wide range of downstream productive activities, including information production, innovation, and the development of products and services, as well as education, community building and interaction, democratic participation, socialization, and many other socially valuable activities. (emphasis added)
Informational resources (especially openly licensed ones) facilitate a wide range of downstream productive activities, including the creation of educational resources. Or, to use language I have used often before, informational resources are “content infrastructure” upon which instructional designers build when they create educational resources.
LLMs will dramatically increase the speed of creating the informational resources that comprise the content infrastructure. Of course the drafts of these informational resources will need to be reviewed and improvements will need to be made – just as is the case with all first drafts – to insure accuracy and timeliness. But it appears that LLMs can get us 80% or so of the way to reasonable first drafts orders of magnitude faster, eliminating the majority of the expense involved in this part of the process. Here’s an example of what I’m talking about. Imagine you’re a SME who has been tasked with writing the content for an introductory economics textbook. (The following examples are from ChatGPT.)
That’s not quite right, but it’s far faster to feed a prompt into ChatGPT and then edit the output to make it accurate than it would be to write that from scratch. And I think this will always be the way to think about these tools – incredibly helpful for creating first drafts for humans to then review, augment, and polish.
But LLMs won’t just help SMEs – they can also provide instructional designers with first drafts of some of the work they do. Imagine you’re an instructional designer who’s been paired with a faculty member to create a course in microeconomics. These tools might help you quickly create first drafts of:
Again, not exactly right, but an incredibly helpful starting point. Especially if you need to create several thousand assessment items.
As we see above, LLMs can engage at a basic level with concepts from the instructional design domain like discussion questions, rubrics, multiple choice questions, and feedback. But in the same way that a SME in a discipline needs to check a draft informational resource for accuracy of information, an instructional designer will need to check these educational resources for pedagogical and psychometric accuracy. And improvements will absolutely need to be made.
In his application to register a work he created using AI software like Stable Diffusion, Steven Thaler wrote that the work “was autonomously created by a computer algorithm running on a machine” and that he was “seeking to register this computer-generated work as a work-for-hire to the owner.” In other words, he applied for copyright protection for a work he created by providing a prompt to a generative AI tool. The US Copyright Office rejected his attempt to register copyright in the work – twice. In their final response they wrote:
Copyright law only protects “the fruits of intellectual labor” that “are founded in the creative powers of the [human] mind.” COMPENDIUM (THIRD) § 306 (quoting Trade-Mark Cases, 100 U.S. 82, 94 (1879)); see also COMPENDIUM (THIRD) § 313.2 (the Office will not register works “produced by a machine or mere mechanical process” that operates “without any creative input or intervention from a human author” because, under the statute, “a work must be created by a human being”)….
While the [review] Board is not aware of a United States court that has considered whether artificial intelligence can be the author for copyright purposes, the courts have been consistent in finding that non-human expression is ineligible for copyright protection….
Courts interpreting the Copyright Act, including the Supreme Court, have uniformly limited copyright protection to creations of human authors…. For this reason, the Compendium of U.S. Copyright Office Practices — the practice manual for the Office — has long mandated human authorship for registration…. Because copyright law as codified in the 1976 Act requires human authorship, the Work cannot be registered.
In other words, as far as the US Copyright Office is concerned, output from programs like ChatGPT or Stable Diffusion are not eligible for copyright protection. Now, that could change if Congress gets involved (are they actually capable of doing anything?), or if the Supreme Court turned its collective back on decades of precedent (which, admittedly, has been happening recently). But unless something rather dramatic happens along these lines, the outputs of generative AI programs will continue to pass immediately into the public domain. Consequently, they will be open educational resources under the common definition:
Open educational resources (OER) are teaching, learning, and research materials that are either (1) in the public domain or (2) licensed in a manner that provides everyone with free and perpetual permission to engage in the 5R activities.
Generative AI tools could have an incredible impact on the breadth and diversity of OER that exist, since they will dramatically decrease the cost and time necessary to create the informational resources that underlie these learning materials. Current funding for the creation of OER (when it’s available at all) typically focuses on the courses enrolling the largest number of students. This makes sense when your theory of philanthropy is that the money you spend should benefit the most people possible. But that theory of philanthropy also means that the “long tail” of courses that each enroll relatively few students are unlikely to ever receive funding. LLMs will radically alter the economics of creating OER, and should make it possible for OER to come to these courses as well. (And while LLMs will have a significant impact on the economics of creating OER, they may not have as dramatic an impact on the sustainability, maintenance, and upkeep of OER over time.)
Two potentially interesting questions: First, in the same way teachers are worried about students submitting work written by LLMs, the same issue is coming for the US Copyright Office. If the output of these tools are not eligible for protection, the Office will soon have a deep interest in understanding which works are created by machines and which are truly the result of human creativity.
Second, how much time and effort would need to be invested in improving the drafts created by one of these tools before the improved version would be recognized as a derivative work that is eligible for copyright protection? (Think about Pride and Prejudice and Zombies. While it was based on a public domain work, it was transformed sufficiently to deserve its own copyright protection.)
I’ve written before that OER advocates would eventually be sorry that they based their advocacy almost exclusively on cost savings, because the price of proprietary learning materials will come down and that talking point will evaporate:
[While] the cost of traditionally copyrighted educational materials has historically been much higher than the price of OER, the cost of textbooks has plateaued for the first time in decades (Perry 2020). As publishers respond to the price pressure created by OER in the course materials market, the difference in the prices of OER and traditionally copyrighted resources is likely to continue to decrease. If the access hypothesis holds, the impact of OER on student outcomes attributable to affordability will decrease in parallel. In other words, adopting OER may not be a long-term strategy for saving students significant amounts of money or closing the achievement gap between lower income and higher income students. (2021)
The impact of LLMs on the economics of creating proprietary materials should be the same as it is for OER (assuming publisher adapt to this new reality). As “big name authors” become supplanted by “big name editors” of work initially drafted by LLMs, the amount of royalties publishers owe to authors will either decrease significantly or may even disappear altogether. Now able to produce and sell proprietary material much less expensively than they were in the past, publishers could either pass this savings on to students in order to compete even more effectively with the price of OER, or simply use the savings to improve their margins, or a little of both. The key point here is that the boon of LLMs for learning materials production is NOT exclusive to OER. In fact, the benefit may accrue disproportionately to publishers of proprietary materials since the publishers of OER have never had to pay royalties. LLMs may put proprietary materials on equal footing with OER in this regard.
We’ve been saying for more than a decade that content is a commodity. That is more true in the context of LLMs than it ever has been before. If content doesn’t differentiate learning materials from one another, what will? It’s possible we may finally be reaching a time when effectiveness will come to the forefront as the primary difference between learning materials. And while content accuracy is the domain of SMEs, effectiveness at supporting learning is the domain of instructional designers.
As the role of SMEs changes from author to editor and their time commitment to projects decreases, the role of instructional designers will grow in importance and effort. The instructional designer is likely the primary prompt engineer. Instructional design expertise will be reflected in the output of these systems in proportion to the degree that instructional design expertise is embedded in the prompts fed into the systems. IDs will engineer prompts and feed them into systems and then do rounds of rapid review of outputs with SMEs at each step in the process from the beginning (what is missing from this list of learning outcomes?) to the end (does the feedback on this formative assessment item accurately correct the misunderstanding a student likely has when they select option B?). As accuracy is being assured via this process, a significant amount of instructional design expertise will need to be applied to the varied outputs of these systems to bring them together in a cohesive, coherent way that will effectively support learning. Again, the right way to think about these tools is that they are incredibly helpful for creating first drafts for humans to then review, augment, and polish. And, when it comes to creating learning materials that are highly effective for all learners, there will be plenty of augmenting and polishing to do.
What an incredible time to be an instructional designer! Every school that teaches instructional design needs to immediately update their curriculum to leverage the existence of these tools. Using LLMs and other AI effectively (e.g., creating custom images) will be a key part of preparing instructional designers for the next decade – a decade that looks to be an absolutely incredible time to be an instructional designer!
I missed this when it was first published, but a few months ago Ithaka S+R shared the results of their Ithaka S+R US Faculty Survey 2021. The survey went out to 145,099 randomly selected faculty members across the US and over 5% of invitees responded. I was particularly interested in the survey’s findings about the state of open educational resources in US higher education. Below, I’ll share these results, the authors’ interpretations of the results, and some of my own thoughts about the results. (The images and quotes from the survey are (C) Ithaka S+R, 2022 and are licensed CC BY-NC.)
The faculty survey asked the following question in 2018 and again in 2021:
Which, if any, of the following open educational resources have you created and/or used in your courses? Please check all that apply.
The authors then explain these results as follows:
There has been a notable increase in the amount of faculty creating and using OER since 2018 (see Figure 39). Overall, more faculty members use OER than create them, though the share of faculty who have created any OER increased has also increased [sic] along with the share of faculty who have used OER. In 2021, 41 percent of faculty reported using open textbooks, 38 percent have used open video lectures, and 26 percent have used open course modules (see Figure 39).
Given the amount of video on the internet (e.g., YouTube) which is not “open” as that word is used in the phrase “open educational resources”, I find it HIGHLY unlikely that almost 40% of faculty found and used videos that were either in the public domain or which were licensed under a CC license granting users the 5R permissions. Which means that when faculty answered this question they were probably actually answering the question “which of the following free online resources have you created and/or used in your courses?” And honestly, which was more likely to happen during the emergency shift to online learning caused by the pandemic – faculty were finding and incorporating any reasonable quality free resource they could find online? Or faculty were taking the time to carefully vet the licensing of the resources they found and specifically chose OER to include in their courses?
In other words, I think this question likely dramatically overestimates that actual amount of OER use happening in higher education, if by “OER use” you mean (as I believe most OER advocates do) “OER adopted as the primary learning resource instead of an expensive proprietary resource.”
It’s also the case that, given the way the question is worded, using a chapter from an open textbook as a supplementary reading in a course where you adopted a $200 textbook would result in a “yes” answer under “open textbooks” here. Which would lead to an even greater overestimation of OER use. I believe a more informative version of the question would have including language like “as the primary required learning resource.”
I found these questions asked in 2018 and again in 2021 even more interesting:
Figure 40. Please read the following statements and indicate the degree to which you agree or disagree with each.
The authors then explain these results as follows:
Unintuitively, while more faculty are using and creating OER since 2018, interest in using and creating OER has declined in that same period. In 2018, 57 percent agreed they were interested in using OER compared to 51 percent in 2021. We see a similar decrease for those interested in creating and publishing OER–30 percent of faculty in 2018 were interested in creating OER in 2018 compared to 25 percent in 2021 (see Figure 40). This may be due to the continued lack of incentive, either monetary or through new professional development opportunities, for adopting OER into their courses.
Note that Question 39 did not, in fact, ask faculty if they “are using” OER; it asked if they “have used” OER. That distinction is incredibly important, because I believe it points to another possible explanation for the change in level of interest the survey found in using OER (whether that means actual OER or simply free online resources). Some of the faculty who adopted and used those resources simply aren’t interested in doing it again. Compared to the full suite of supports and helps they’re used to getting from commercial publishers – not just the textbook content, but thousands of assessment items, Powerpoints to use in class, pacing guides for 15 and 8-week terms, analytics that help faculty track struggling students, etc. – the overwhelming majority of OER offer very little support for faculty. It is, objectively, more work for faculty when they switch from adopting a full courseware solution to using a free PDF. For example, if you switched from MyMathLab to an OpenStax book for college algebra, you would find yourself hand-grading a lot of homework the courseware used to grade for you. It’s not hard to understand why, having experienced all that additional manual work once, faculty would no longer be interested in adopting OER in the future. That’s how rates of “have used” OER can go up, while rates of “interested in using OER” can go down.
The authors’ explanation of why interest in adopting OER has dropped – that faculty aren’t being paid extra to adopt them – is itself an implicit acknowledgement of how much extra work it is for faculty to adopt and use OER in their classes. No one has to provide incentives for faculty to adopt publisher materials for their classes – faculty are intrinsically motivated to adopt those resources because they save faculty time and effort. Demanding extra pay before adopting OER only makes sense if you believe it will take extra work to adopt and use OER (compared to adopting and using publisher materials).
Of course it’s possible to adopt OER within the courseware context, integrated with the complete set of supports faculty want, but many OER advocates reject that way of using OER because there are some costs associated with it.
In years to come I expect that interest in adopting and using “naked OER” – just the openly licensed content – will continue to decline until only early adopter-types are left. The only real hope I see for a future in which OER adoption goes to scale within US higher education is one in which OER are integrated and combined with all the other helps and supports faculty want. Because faculty decide what gets adopted and used in class.
FreshRSS 