(57) Evaluation of student assignment on artificial intelligence and its utility in answering drug information questions

Objective : Evaluate the use of an artificial intelligence assignment to instruct students on the accuracy of ChatGPT to answer consumer drug information questions in a Foundations in Pharmacy Practice course.

Methods: The Foundations in Pharmacy Practice series in the curriculum includes a second-year course with material related to pharmaceutical calculations and drug information practice. Instructors requested ChatGPT answers with references to common consumer questions and saved as PDF files. The students were assigned a PDF to evaluate and write a professional drug information response. The exercise template included an overall assessment on a Likert scale of 1 (poor) to 5 (excellent). The form included whether the information was complete and if they could find the references cited by ChatGPT. The students were required to provide a tertiary and primary source to answer the question and compare their results with ChatGPT. Each student had to provide an overall assessment. Descriptive statistics were utilized.

Results: Of the 76 students, the median score of the Likert scale was 3 indicating average quality of the ChatGPT response. Only 4% of students were able to find the citations provided by ChatGPT to answer the question. All students were able to find both a tertiary and a primary source to provide a professional answer to the consumer. Only 20% of the responses were considered accurate with another 28% being correct, but incomplete in providing a sufficient answer. Misleading or incorrect answers constituted 52% of responses.

Conclusions: The ChatGPT exercise was a novel approach to informing students about the use of artificial intelligence in answering consumer drug information and demonstrated the utility and challenges of some artificial intelligence.