Performance of large language models on Thailand's national medical licensing examination: a cross-sectional study
4
Issued Date
2025-01-01
Resource Type
eISSN
19755937
Scopus ID
2-s2.0-105005377591
Pubmed ID
40354784
Journal Title
Journal of educational evaluation for health professions
Volume
22
Rights Holder(s)
SCOPUS
Bibliographic Citation
Journal of educational evaluation for health professions Vol.22 (2025) , 16
Suggested Citation
Saowaprut P., Wabina R.S., Yang J., Siriwat L. Performance of large language models on Thailand's national medical licensing examination: a cross-sectional study. Journal of educational evaluation for health professions Vol.22 (2025) , 16. doi:10.3352/jeehp.2025.22.16 Retrieved from: https://repository.li.mahidol.ac.th/handle/123456789/110365
Title
Performance of large language models on Thailand's national medical licensing examination: a cross-sectional study
Author(s)
Author's Affiliation
Corresponding Author(s)
Other Contributor(s)
Abstract
PURPOSE: This study aimed to evaluate the feasibility of general-purpose large language models (LLMs) in addressing inequities in medical licensure exam preparation for Thailand's National Medical Licensing Examination (ThaiNLE), which currently lacks standardized public study materials. METHODS: We assessed 4 multi-modal LLMs (GPT-4, Claude 3 Opus, Gemini 1.0/1.5 Pro) using a 304-question ThaiNLE Step 1 mock examination (10.2% image-based), applying deterministic API configurations and 5 inference repetitions per model. Performance was measured via micro- and macro-accuracy metrics compared against historical passing thresholds. RESULTS: All models exceeded passing scores, with GPT-4 achieving the highest accuracy (88.9%; 95% confidence interval, 88.7-89.1), surpassing Thailand's national average by more than 2 standard deviations. Claude 3.5 Sonnet (80.1%) and Gemini 1.5 Pro (72.8%) followed hierarchically. Models demonstrated robustness across 17 of 20 medical domains, but variability was noted in genetics (74.0%) and cardiovascular topics (58.3%). While models demonstrated proficiency with images (Gemini 1.0 Pro: +9.9% vs. text), text-only accuracy remained superior (GPT-4o: 90.0% vs. 82.6%). CONCLUSION: General-purpose LLMs show promise as equitable preparatory tools for ThaiNLE Step 1. However, domain-specific knowledge gaps and inconsistent multi-modal integration warrant refinement before clinical deployment.