Login
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the United States Medical Licensing Examination (USMLE), a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study calibration of the probabilities, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively by presenting a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.
Here is the full paper by Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvita. Ho hum, people, ho hum!
Via Ethan Mollick.
The post Capabilities of GPT-4 on Medical Challenge Problems appeared first on Marginal REVOLUTION.
When the next pandemic sweeps the United States, health officials in Ohio won’t be able to shutter businesses or schools, even if they become epicenters of outbreaks. Nor will they be empowered to force Ohioans who have been exposed to go into quarantine. State officials in North Dakota are barred from directing people to wear masks to slow the spread. Not even the president can force federal agencies to issue vaccine or testing mandates to thwart its march.
Conservative and libertarian forces have defanged much of the nation’s public health system through legislation and litigation as the world staggers into the fourth year of covid.
At least 30 states, nearly all led by Republican legislatures, have passed laws since 2020 that limit public health authority, according to a Washington Post analysis of laws collected by Kaiser Health News and the Associated Press as well as the Association of State and Territorial Health Officials and the Center for Public Health Law Research at Temple University.
Health officials and governors in more than half the country are now restricted from issuing mask mandates, school closures, and other protective measures or must seek permission from their state legislatures before renewing emergency orders, the analysis showed.
Here is more from the Washington Post.
The post Restrictions on state public health authorities appeared first on Marginal REVOLUTION.
…we often discuss mental health in terms of treatment and selection effects. While more causal inference is needed, I believe it some points are often overlooked.
Personality plays a role: Many in the field can be characterized as overachievers. This behavior can easily turn pathological if it is driven by a fear of failure or a sense that self-worth is contingent on competence. Moreover in a competitive academic environment. Exit may be psychologically very difficult if your self-worth is on the line.
Policies within graduate programs exacerbate the issue: In my program, if a student drops out, the University will not award them a master’s degree if they already have a similar degree from another university. This policy discourages students exit and may keep them in situations that are not beneficial for their mental health.
Economists tend to overrate the effectiveness of educational signals in selecting prospective grad study: Interviews are often not a part of the selection process, which I believe is a missed opportunity to assess a student’s psychological readiness for a PhD program. For many far less stressful jobs psychological testing is standard. In my experience, I only received interviews from programs that had already accepted me (meant to convince me to accept offers).
From anonymous.
The post On graduate student mental health (from my email) appeared first on Marginal REVOLUTION.
I am no expert on this weight loss drug, but many people on both the bio side and the VC side are telling me it works. Might it, or some variant thereof, become the best-selling drug of all time?
Just think that within five years we likely will have come up with good, serious remedies to Covid, to obesity (a major, major public health problem, especially in America), to malaria, and to dengue, vaccines in the latter two cases. And that is unlikely to be the end of the list.
That is an astonishing record, and we are truly living in a golden age for biomedicine. Ozempic is further evidence that the great stagnation is over, even though the current world is not mimicking the physical dynamism of say the 1920s.
The post Semaglutide, Ozempic, and the end of the Great Stagnation appeared first on Marginal REVOLUTION.
Shakked Noy, MIT economics, to do RCTs on GPTs as teaching and learning tools.
Gabriel Birnbaum, Bay Area, from Fortaleza, Brazil, to investigate lithography as a key technology used in the manufacturing of microchips.
Moritz Wallawitsch, Berkeley. RemNote is his company, educational technology, and to develop a complementary podcast and for general career development.
Katherine Silk, Boston/Cambridge, general career support and to support advice for early-stage startups.
Benjamin Schneider, Brooklyn. To write a book on the new urbanism.
Joseph Walker, Sydney, Australia, to run and expand the Jolly Swagman podcast.
Avital Balwit, Bay area, travel grant and general career development.
Benjamin Chang, Cambridge, MA. General career support, “I will develop novel RNA riboswitches for gene therapy control in human cells using machine learning.”
Daniel Kang, Berkeley/Champagne-Urbana, biometrics and crypto.
Aamna Zulfifiqar, Karachi, Pakistan, to attend UK higher education to study economics.
Jeremy Stern, Glendale, CA, Tablet magazine. To write a book.
James Meech, PhD student, Cambridge, UK, to work on a random number generator for better computer architectures.
Arthur Allshire, University of Toronto, background also in Ireland and Australia, robotics and support to attend conferences.
Jason Hausenloy, 17, Singapore, travel and general career development, issues surrounding artificial intelligence.
Sofia Sanchez, Metepec, Mexico, biology and agricultural productivity, to spend a summer at a Stanford lab.
Ukraine tranche:
Andrey Liscovich, eastern Ukraine, formerly of Harvard, to provide equipment for public transportation, communication and emergency power generation to civilian authorities of frontline-adjacent areas in Ukraine which have lost vital infrastructure.
Chris Nicholson, Bay area, working as a broker to maintain internet connectivity in Ukraine.
Andrii Nikolaiev, Arsenii Nikolaiev, Zarina Kodyrova, Kvanta, to advance Ukrainian mathematics, help and train math Olympiad winners.
As usual, India and Africa/Caribbean tranches will be reported separately.
The post Emergent Ventures winners, 24th cohort appeared first on Marginal REVOLUTION.
A new drug quashes all coronavirus variants. But regulatory hurdles and a lack of funding make it unlikely to reach the U.S. market anytime soon.
So starts the NYT article. Have we learned nothing? As for the drug itself, the news is good:
…a new class of variant-proof treatments could help restock the country’s armory. Scientists on Wednesday reported in The New England Journal of Medicine that a single injection of a so-called interferon drug slashed by half a Covid patient’s odds of being hospitalized.
The results, demonstrated in a clinical trial of nearly 2,000 patients, rivaled those achieved by Paxlovid. And the interferon shots hold even bigger promise, scientists said. By fortifying the body’s own mechanisms for quashing an invading virus, they can potentially help defend against not only Covid, but also the flu and other viruses with the potential to kindle future pandemics.
We will see if this has the opportunity to progress. Fast Grants, working in conjunction with Rainwater Foundation, was a key early funder here.
Oh, and do note this:
As it stands, Eiger executives said that they might seek authorization for the interferon shot outside of the United States. China, for example, has been looking for new treatment options.
And here is an Eric Topol thread on the results.
The post Pegylated interferon lambda appeared first on Marginal REVOLUTION.
Manifold Markets is currently putting the chance at 12%. That seems high to me, but maybe two percent? Which I still regard as leading to a high expected cost, to be clear. There is now mammal-to-mammal transmission. Zeynep Tufekci considers possible preparatory steps (NYT), including more surveillance.
The post Will there by an H5N1 pandemic for humans? appeared first on Marginal REVOLUTION.
Or will they be thwarted?
To Nina McCollum, Cleveland Clinic’s decision to begin billing for some email correspondence between patients and doctors “was a slap in the face.”
She has relied on electronic communications to help care for her ailing 80-year-old mother, Penny Cooke, who is in need of specialized psychiatric treatment from the clinic. “Every 15 or 20 dollars matters, because her money is running out,” she said.
Electronic health communications and telemedicine have exploded in recent years, fueled by the coronavirus pandemic and relaxed federal rules on billing for these types of care. In turn, a growing number of health care organizations, including some of the nation’s major hospital systems like Cleveland Clinic, doctors’ practices and other groups, have begun charging fees for some responses to more time-intensive patient queries via secure electronic portals like MyChart.
…a new study shows that the fees, which some institutions say range from a co-payment of as little as $3 to a charge of $35 to $100, may be discouraging at least a small percentage of patients from getting medical advice via email. Some doctors say they are caught in the middle of the debate over the fees, and others raised concerns about the effects that the charges might have on health equity and access to care.
Demand curves do slope downward. And yet:
But a recent study led by Dr. Holmgren of data from Epic, a dominant electronic health records company, showed that the rate of patient emails to providers had increased by more than 50 percent in the last three years.
Perhaps there is a smidgen of room for AI here? But not under the current legal regime, I suspect. Here is the full Benjamin Ryan NYT article.
The post Medical markets in everything? appeared first on Marginal REVOLUTION.
Universities have been told they must limit the number of medical school places this year or risk fines, a move attacked as “extraordinary” when the NHS is struggling with staff shortages.
Medical schools have been told to curtail offers to ensure that there is “no risk” of them accepting more would-be doctors than permitted by a government cap, with universities saying they are likely to offer fewer places than normal to sixth-formers this year.
Ministers have been criticised for holding firm to a 7,500 cap on new medical students in England while also acknowledging that a chronic shortage of doctors and nurses is contributing to long delays for NHS treatment.
Robert Halfon, the universities minister, wrote to vice-chancellors last week telling them to limit their offers to sixth-formers, causing frustration among universities, which face fines of £100,000 per student for persistent over-recruitment. Universities say that in the summer, they were forced to reject students for administrative reasons such as submitting vaccine certificates late to stay within permitted numbers.
Here is more from the Times of London (gated). Perhaps Tyrone approves!
The post The medical culture that is Britain appeared first on Marginal REVOLUTION.
FreshRSS