Large language model triaging of simulated nephrology patient inbox messages
Issued Date
2024-01-01
Resource Type
eISSN
26248212
Scopus ID
2-s2.0-85204728264
Journal Title
Frontiers in Artificial Intelligence
Volume
7
Rights Holder(s)
SCOPUS
Bibliographic Citation
Frontiers in Artificial Intelligence Vol.7 (2024)
Suggested Citation
Pham J.H., Thongprayoon C., Miao J., Suppadungsuk S., Koirala P., Craici I.M., Cheungpasitporn W. Large language model triaging of simulated nephrology patient inbox messages. Frontiers in Artificial Intelligence Vol.7 (2024). doi:10.3389/frai.2024.1452469 Retrieved from: https://repository.li.mahidol.ac.th/handle/20.500.14594/101418
Title
Large language model triaging of simulated nephrology patient inbox messages
Corresponding Author(s)
Other Contributor(s)
Abstract
Background: Efficient triage of patient communications is crucial for timely medical attention and improved care. This study evaluates ChatGPT’s accuracy in categorizing nephrology patient inbox messages, assessing its potential in outpatient settings. Methods: One hundred and fifty simulated patient inbox messages were created based on cases typically encountered in everyday practice at a nephrology outpatient clinic. These messages were triaged as non-urgent, urgent, and emergent by two nephrologists. The messages were then submitted to ChatGPT-4 for independent triage into the same categories. The inquiry process was performed twice with a two-week period in between. ChatGPT responses were graded as correct (agreement with physicians), overestimation (higher priority), or underestimation (lower priority). Results: In the first trial, ChatGPT correctly triaged 140 (93%) messages, overestimated the priority of 4 messages (3%), and underestimated the priority of 6 messages (4%). In the second trial, it correctly triaged 140 (93%) messages, overestimated the priority of 9 (6%), and underestimated the priority of 1 (1%). The accuracy did not depend on the urgency level of the message (p = 0.19). The internal agreement of ChatGPT responses was 92% with an intra-rater Kappa score of 0.88. Conclusion: ChatGPT-4 demonstrated high accuracy in triaging nephrology patient messages, highlighting the potential for AI-driven triage systems to enhance operational efficiency and improve patient care in outpatient clinics.