(16) Assessing ChatGPT's Response Accuracy in Pharmacy Education: Insights into Cognitive Complexity

Objective : To evaluate the accuracy of ChatGPT’s responses in pharmacy education using standardized exam questions.

Methods: One hundred twenty exam questions based on the NAPLEX study guide covering biostatistics, calculation, and therapeutics (40 questions per subject) were administered. The questions were evenly distributed in four main levels of Bloom’s Taxonomy including Recall, Understand, Apply, and Analyze. The responses from ChatGPT-3.5 were classified into three categories: correct (similar to the provided answers), incorrect (different from the provided answers), or partially correct (partially matched with the provided answers).

Results: The result showed that ChatGPT's responses were 65% (n = 78) correct, 29.17% (n = 35) incorrect, and 5.83% (n = 7) partially correct. Particularly, biostatistics has 70% (n = 28) accuracy, calculation 65% (n = 26), and therapeutics the lowest at 60% (n = 24). Accuracy varied across Bloom's Taxonomy, from 76.67% (n = 23) in Recall and 73.33% (n = 22) in Understand to 60% (n = 18) in Apply and 50% (n = 15) in Analyze. This indicates a trend where accuracy dipped with increasing question complexity. Similarly, within the correct responses, the accuracy declined from basic questions in Recall (29.49%, n = 23) to higher-order cognitive tasks in Analyze (19.23%, n = 15). The data suggests that while ChatGPT is reliable for simpler questions, its accuracy decreases with more complex analytical cases.

Conclusions: The study reveals ChatGPT as a potential tool in pharmacy education, demonstrating a 65% accuracy rate across a variety of exam questions. However, its accuracy declines as the complexity of questions increases with the level in Bloom's Taxonomy. Our future study will expand the framework into the entire pharmacy program curriculum.