(63) Exploring the Role of ChatGPT in Enhancing Multiple-Choice Questions for Pharmacy Education

Objective : Recent AI advancements have revolutionized various medical fields, with tools like ChatGPT excelling on complex tasks and high-stakes exams. However, research on AI applications in pharmacy assessment is limited, presenting an unexplored opportunity. The objective of this research is to evaluate ChatGPT's effectiveness as a tool for revising flawed multiple-choice exam questions in pharmacy education, exploring its potential to streamline the revision process, address common flaws, and support faculty development.

Methods: This study evaluated the use of ChatGPT (GPT-4 model) for revising flawed multiple-choice exam questions in a P2 pharmacy course. ChatGPT revisions employed zero-shot prompts specifying parameters like Bloom's Taxonomy level, target knowledge, and a request for plausible distractors. The original item writers reviewed these revisions and provided feedback. Any unacceptable changes were manually corrected through collaboration between the writers and the review team.

Results: Of 94 items, 35% were retained without edits. GPT-4 successfully generated 12 new items from unfocused stems and aided in revising answer choice flaws for 15 items. However, GPT-4 failed to generate viable replacements for some unfocused stems due to items becoming too narrow or exceeding course scope. Additionally, several AI-generated answer choices were rejected as irrelevant or incorrect. Despite these failures, rejected GPT-4 output often served as a template for successful manual revisions.

Conclusions: ChatGPT successfully revised unfocused stems and reworded text for clarity, but sometimes misaligned with the original intent or produced inaccurate 'hallucinations.' However, even rejected ChatGPT output offered useful item formatting templates. These findings highlight the need for human experts to review AI-generated revisions but also suggest that AI can play a role in guiding faculty development for item writing.