Mystery AI Hype Theater 3000

Take a deep breath and join Alex and Emily in AI Hell itself, as they take down a month's worth of hype in a mere 60 minutes.This episode aired on Friday, May 5, 2023.Watch the video of this episode on PeerTube.References:Terrifying NEJM article on GPT-4 in medicine“Healthcare professionals preferred ChatGPT 79% of the time”Good thoughts from various experts in responseChatGPT supposedly reading dental x-raysChatbots “need” therapistsCEO proposes AI therapist, removes proposal upon realizing there’s regulation:https://twitter.com/BEASTMODE/status/1650013819693944833 (deleted)ChatGPT is more carbon efficient than human writersAsking disinformation machine for confirmation biasGPT-4 glasses to tell you what to say on dates, "Charisma as a Service"Context-aware fill for missing data“Overemployed” with help from ChatGPTPakistani court uses GPT-4 in bail decisionChatGPT in Peruvian and Mexican courtsElon Musk’s deepfake defenseElon Musk's TruthGPTFake interview in German publication revealed as “AI” at the end of the articleYou can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

After a hype-y few weeks of AI happenings, Alex and Emily shovel the BS on GPT-4’s “system card,” its alleged “sparks of Artificial General Intelligence,” and a criti-hype heavy "AI pause" letter. Hint: for a good time, check the citations.This episode originally aired on Friday, April 7, 2023.You can also watch the video of this episode on PeerTube.References:GPT-4 system card: https://cdn.openai.com/papers/gpt-4-system-card.pdf“Sparks of AGI” hype: https://twitter.com/SebastienBubeck/status/1638704164770332674And the preprint from Bubeck et al.: https://arxiv.org/abs/2303.12712“Pause AI” letter: https://futureoflife.org/open-letter/pause-giant-ai-experiments/The “Sparks” paper points to this 1997 editorial in their definition of “intelligence”:https://www1.udel.edu/educ/gottfredson/reprints/1997mainstream.pdfRadiolab's miniseries, 'G': https://radiolab.org/series/radiolab-presents-gBaria and Cross, "The brain is a computer is a brain.": https://arxiv.org/abs/2107.14042Senator Chris Murphy buys the hype:https://twitter.com/ChrisMurphyCT/status/1640186536825061376Generative “AI” is making “police sketches”:https://twitter.com/Wolven/status/1624299508371804161?t=DXyucCPYPAKNn8TtAo0xeg&s=19More mathy math in policing:https://www.cbsnews.com/colorado/news/aurora-police-new-ai-system-bodycam-footage/?utm_source=dlvr.it&utm_medium=twitterUser Research without the Users:https://twitter.com/schock/status/1643392611560878086DoNotPay is here to cancel your gym membership:https://twitter.com/BrianBrackeen/status/1644193519496511488?s=20You can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Alex and Emily are taking AI to court! Amid big claims about LLMs, a look at the facts about ChatGPT, legal expertise, and what the bar exam actually tells you about someone's ability to practice law--with help from Harvard legal and technology scholar Kendra Albert.This episode was first recorded on March 3, 2023.Watch the video of this episode on PeerTube.References:Social Science Research Network paper “written” by ChatGPTJoe Wanzala, “ChatGPT is ideal for eDiscovery”Legal applications for ChatGPT:Shot: GPT-4 'could pass the. bar exam'Chaser: ChatGPT had bigger dreams. "AI for law"“AI can legally run a company”Wired: Generative AI Is Coming for the Lawyers“This is a decision by a Colombian court in Cartagena (dated January, 30, 2023).As far as we know, it is the first time that a judicial decision has been taken by explicitly resorting to #ChatGPT @sama @OpenAI. The Court poses a series of specific questions to #ChatGPT""Don't Be A Lawyer" song from "Crazy Ex-Girlfriend"Rep. Ted Lieu introduces legislation written by an LLMDoNotPay offers money to anyone willing to use their AI to argue in courtFresh AI Hell:Vanderbilt University responds to MSU shooting with e-mail written using ChatGPTScience fiction magazine closes submissions due to LLM spamThe 1st International Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT’23)You can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Should the mathy-maths be telling doctors what might be wrong with you? And can they actually help train medical professionals to treat human patients? Alex and Emily discuss the not-so-real medical and healthcare applications of ChatGPT and other large language models.Plus another round of fresh AI hell, featuring "charisma as a service," and other assorted reasons to tear your hair out.This episode was first recorded on February 17th of 2023.Watch the video of this episode on PeerTube.References:Glass.ai makes “diagnosis machine”:https://twitter.com/AiBreakfast/status/1620128621821317125?t=Q6tTAOcGAoFJ3Ko9m4EC9g&s=19Percy Liang claims 'PubMedGPT' can pass medical exams:https://crfm.stanford.edu/2022/12/15/pubmedgpt.htmlhttps://twitter.com/percyliang/status/1603469265583353856?s=20&t=SdWeINzUw92pbkTO8OAVqQEmily's reaction to the above:https://twitter.com/emilymbender/status/1603766381807570944?s=20&t=SdWeINzUw92pbkTO8OAVqQChatGPT gets 60 percent of questions right in US Medical Licensing Exam:https://healthitanalytics.com/news/chatgpt-passes-us-medical-licensing-exam-without-clinician-inputAn Apple Watch error is clogging up 911 lines:https://www.nytimes.com/2023/02/03/health/apple-watch-911-emergency-call.htmlChatGPT-assisted diagnosis: Is the future suddenly here?https://www.statnews.com/2023/02/13/chatgpt-assisted-diagnosis/NVIDIA “eye contact” demo:https://twitter.com/Jousefm2/status/1616878021280993284“Theory of the mind":https://twitter.com/LChoshen/status/1623575423652139015?t=Ohc9tzB09pAEddAReLc6mA&s=09You can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

New year, new hype? As the world gets swept up in the fervor over ChatGPT of late 2022, Emily and Alex give a deep sigh and begin to unpack the wave of fresh enthusiasm over large language models and the "chat" format specifically.Plus, more fresh AI hell.This episode was recorded on January 20, 2023.Watch the video of this episode on PeerTube.References:Situating Search (Shah & Bender 2022) Related op-ed: https://iai.tv/articles/all-knowing-machines-are-a-fantasy-auid-2334Piantadosi's thread showing ChatGPT writing a program to classify white males as good scientistsFind Anna Lauren Hoffman's publications (though not yet the one we were referring to) here: https://www.annaeveryday.com/publicationsSarah T. Roberts, Behind the Screen Karen Hao's AI Colonialism series Milagros Miceli: https://www.weizenbaum-institut.de/en/spezialseiten/persons-details/p/milagros-miceli/Julian Posada: https://posada.website/“This Isn’t Your Data, Friend”: Black Twitter as a Case Study on Research Ethics for Public Data (Klassen & Fiesler 2022) No Humans Here: Ethical Speculation on Public Data, Unintended Consequences, and the Limits of Institutional Review (Pater, Fiesler & Zimmer 2022) Casey Fiesler's publications: https://caseyfiesler.com/publications/And TikTok: https://www.tiktok.com/@professorcaseyWhere are human subjects in Big Data research? The emerging ethics divide. (Metcalf & Crawford 2016) You can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Emily and Alex are joined by Dr. Jeremy G. Kahn to discuss the distressingly large number of evaluation metrics for artificial intelligence, and some new AI hell.Jeremy G. Kahn has a PhD in computational linguistics, with a focus on information-theoretic and empirical engineering approaches to dealing with natural language (in text and speech). He’s gregarious, polyglot, a semi-auto-didact, and occasionally prolix. He also likes comic books, coffee, progressive politics, information theory, lateral thinking, science fiction, science fact, linear thinking, bicycles, beer, meditation, love, play, and inquiry. He lives in Seattle with his wife Dorothy and son Elliott.This episode was recorded on December 12, 2022.Watch the video of this episode on PeerTube.References:XKCD: StandardsWikidataConGish GallopThe Bender RuleDJ Khaled - You Played YourselfJeff Kao's interrogation of public comment periods.Emily's blog post response to NYT pieceYou can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Emily and Alex discuss MetaAI's bullshit science paper generator, Galactica, along with its defenders. Plus, where could AI actually help scientific research? And more Fresh AI Hell. Watch the video of this episode on PeerTube. References:Imre Lakatos on research programsShah, Chirag and Emily M. Bender. 2022. Situating Search. Proceedings of the 2022 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’22). UW RAISE (Responsibility in AI Systems and Experiences)Stochastic Parrots:Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜. In Proceedings of FAccT 2021, pp.610-623. The Octopus Paper:Bender Emily M. and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. ACL 2020Palestinian man arrested because of bad machine translation.Katherine McKittrick, Dear Science and Other StoriesThe Sokal HoaxSafiya Noble, Algorithms of OppressionLatanya Sweeney, "Discrimination in Online Ad Delivery"Mehtab Khan and Alex Hanna, The Subjects and Stages of AI Dataset Development: A Framework for Dataset Accountability(What is 'sealioning'?)http://wondermark.com/1k62/Grover:Raji, Inioluwa Deborah, Emily M. Bender, Amandalynne Paullada, Emily Denton and Alex Hanna. 2021. AI and the Everything in the Whole Wide World Benchmark. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.Ben Dickson's coverage of Grover:Why we must rethink AI benchmaYou can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Emily and Alex discuss Sam Bankman-Fried's Future Fund, its essay contest, and the problems with using AI for prediction and resource allocation--mere days before the collapse of FTX. Also, we introduce our "What in the Fresh AI Hell?" segment!This episode was recorded on November 9, 2022. Watch the video of this episode on PeerTube.References:Super Forecasting And AIBill Howe - Applied AI in High-Expertise Settings or Curation as ProgrammingSamir Passi and Solon Barocos - "Problem Formulation in Fairness"Vinod Prabharakan, William Isaac, Donald Martin Jr - Participatory Problem Formation for Fairer Machine LearningDavid Ribes, Andrew S Hoffman, Steven C Slota and Geoffrey C Bowker -The Logic of DomainsShoshana Zuboff - The Age of Surveillance CapitalismLee Vinsel on Criti-hype - Notes on Criticism and Technology HypeYou can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

AI is increasingly being used to make visual art. But when is an algorithmically-generated image art...and when is it just an aesthetically pleasing arrangement of pixels? Technology researchers Emily M. Bender and Alex Hanna talk to a panel of artists and researchers about the hype, the ethics, and even the definitions of art when a computer is involved.This episode was recorded in October of 2022. You can watch the video on PeerTube.Dr. Johnathan Flowers is an assistant professor in the department of philosophy at California State University, Northridge. His research interest is at the intersection of American Pragmatism, Philosophy of Disability, and Philosophy of Race, Gender and Sexuality as they apply to socio-technical systems. Flowers also explores the impacts of cultural narratives on the perception and development of sociotechnical systems.Dr. Jennifer Lena is a professor at Teachers College, Columbia University, where she runs the Arts Administration program. She’s published books on music genres, the legitimation of art, and the measurement of culture.Dr. Negar Rostamzadeh is a Senior Research Scientist at Google Responsible AI team. Her recent research is at the intersection of computer vision and sociotechnical research. She studies creative computer vision technologies and the broader social impact of them. Kevin Roose, "An A.I.-Generated Picture Won an Art Prize. Artists Aren’t Happy."Jo Lawson-Tancred, "Robot Artist Ai-Da Just Addressed U.K. Parliament About the Future of A.I. and ‘Terrified’ the House of Lords"Marco Donnarumma, "AI Art Is Soft Propaganda for the Global North"Jane Recker, "U.S. Copyright Office Rules A.I. Art Can’t Be Copyrighted"Richard Whiddington, "Shutterstock Inks Deal With DALL-E Creator to Offer A.I.-Generated Stock Images. Not All Artists Are Rejoicing."Stephen Cave and Kanta Dihal, "The Whiteness of AI"Follow our guests:Dr. Johnathan Flowers - https://twitter.com/shengokai // https://zirk.us/@shengokaiDr. Negar Rostamzadeh - twitter.com/negar_rzDr. JeYou can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.

Technology researchers Emily M. Bender and Alex Hanna kick off the Mystery AI Hype Theater 3000 series by reading through, "Can machines learn how to behave?" by Blaise Aguera y Arcas, a Google VP who works on artificial intelligence.This episode was recorded in September of 2022, and is the last of three about Aguera y Arcas' post.You can watch the video of this episode on PeerTube.You can check out future livestreams at https://twitch.tv/DAIR_Institute. Follow us!Emily Twitter: https://twitter.com/EmilyMBender Mastodon: https://dair-community.social/@EmilyMBender Bluesky: https://bsky.app/profile/emilymbender.bsky.social Alex Twitter: https://twitter.com/@alexhanna Mastodon: https://dair-community.social/@alex Bluesky: https://bsky.app/profile/alexhanna.bsky.social Music by Toby Menon.Artwork by Naomi Pleasure-Park. Production by Christie Taylor.