Published on
In a study of large language models (LLMs) published in Nature Medicine, researchers found chatbots provide inaccurate and inconsistent information to people seeking medical advice, which could pose health risks. What’s more, consumers did not make better health decisions than those who used traditional sources for their information such as internet searches. The randomized trial asked 1,298 participants to identify recommended actions for 10 medical scenarios (eg, developing a severe headache after a night out, a new mother feeling constantly out of breath)—with the control group using traditional online search for information and the intervention group using an LLM. Tested alone, the LLMs assessed the scenarios accurately, but the human participants often didn’t know what information to provide to the models in their prompts to figure out the best disposition for the health condition, such as going to see a primary care provider or going to the emergency department. In an extreme case, 2 users described similar symptoms of a subarachnoid hemorrhage to the LLM. One user was told to lie down in a dark room, and the other user was given the correct recommendation to seek emergency care.
Doctor chatbot: The use of chatbots for medical advice brings up the larger legal question of whether chatbots are practicing medicine without a license, says Alan A. Ayers, MBA, MAcc, President of Urgent Care Consultants and Senior Editor of JUCM. A life insurance company recently sued OpenAI, alleging ChatGPT engaged in the unauthorized practice of law. According to the complaint, the AI acted as a “legal assistant,” advising a claimant to reopen a settled disability case and helping draft over 40 legal filings. “If an AI model can be sued for crossing the line into unlicensed legal practice, how long until we see a similar test case in healthcare?” Ayers asks. “Patients are increasingly turning to generative AI for symptom checking, triage, and treatment advice instead of visiting a clinic. As AI blurs the lines of professional counsel, we are left with a critical question: When does an algorithm’s output cross the threshold from a harmless web search into the unlicensed practice of medicine?”