Published on
Researchers from Dartmouth found that when using large language models (LLMs) to draft responses to patient portal messages, clinicians may spend more time editing the AI responses than it would take to write them manually in the first place. Using more than 610,000 patient portal messages, the researchers created a training dataset of 144,000 clinician–patient conversations and tested multiple commercial and locally hosted LLMs against expert-written responses. In the study —which was presented at the 64th Annual Meeting of the Association for Computational Linguistics—the best-performing models reduced clinicians’ expected editing workload by 25%–26%, indicating that revisions to the drafted responses were often still needed. The authors conclude that LLMs show promise for drafting patient portal replies, but meaningful gains in efficiency will require customization to individual clinicians’ preferences and communication styles rather than relying on generic AI models.
Work in progress: A separate post from Dartmouth about the study notes that the portals’ automated responses are often too long, don’t ask follow-up questions, and use irrelevant or inaccurate medical details. However, the authors also say that as AI improves, portal messages will likely require less mental energy from clinicians.