(344) Comparison of flawed vs. non-flawed multiple-choice questions as preparatory material for a pharmacotherapeutics exam

Objective : To assess the impact of using flawed and non-flawed multiple-choice questions as study tools on exam performance.

Methods: Using a prospective crossover study design, students enrolled in an Integrated Pharmacotherapeutics I course, in the second-year of pharmacy school, were assigned optional practice questions to use as an examination study tool. Participants were randomly assigned to one of two groups, with each group receiving parallel sets of expert-reviewed practice questions for heart failure (HF) and hypertension (HTN) content, in which one set contained item writing flaws (IWFs, control group) and the other set was free of IWFs (intervention group). Exam scores were compared using MANOVA and Student’s t-test, and Cohen’s d was used to evaluate the effect size.

Results: A total of 59 (93.7%) students engaged in both practice quizzes. There was not a statistically significant difference on total exam performance between students who used practice quizzes containing flawed or unflawed questions. Similarly, there was not a statistically significant difference in performance on the HF or HTN subsection of questions on the exam between the groups. However, for all students, there was a statistically significant difference in exam performance on questions relating to HTN and HF, compared to non-practice topics (p=0.005; effect size 0.52).

Conclusions: The results of this study confirm current pedagogical theory that utilizing practice questions to prepare for examinations improves exam scores. This study failed to demonstrate a significant difference in exam performance between students utilizing flawed and unflawed practice questions. As this is the first study to evaluate the potential impact of flawed versus unflawed practice questions on exam performance, additional research with larger sample sizes is warranted.