(243) Operationalizing “High Quality” in Short Answer Assessment Questions Across a Pharmacy School Division

Objective : Best practices have been published for the construction of short answer assessment questions but there is no tool available to measure the quality of these questions in a systematic manner.

Methods: The authors conducted a literature review on short answer question construction best practices and used these to create an initial quality evaluation tool. Two rounds of testing, discussion, and revisions occurred before arriving at the final tool (20 items). The final tool was then applied to all short answer exam questions in required Doctor of Pharmacy courses within the Social & Administrative Sciences division during the 2022-23 academic year as part of a quality assurance initiative.

Results: Inter-rater agreement using the final tool on a sample of 26 assessment questions was 88.89% prior to discussion and achieving consensus. The areas most frequently contributing to disagreement were whether key words were properly emphasized with bolding or underlining in a question, which details in a question were considered extraneous, and whether the expected length for written answers was stated clearly.

Conclusions: Operationalizing the quality of a short answer question posed unexpected challenges due to the subjectivity of many of the best practices but yielded valuable information to the faculty for future assessments.