(281) The Chatbot Is In: A Qualitative Analysis of Student Experiences with Chatbots for Drug Information

Objective : To explore pharmacy student perceptions of generative artificial intelligence (AI) chatbot responses after the introduction of a pilot assignment into the curriculum of a didactic drug information course.

Methods: Second-year pharmacy students completed an assignment in the fall semester of 2023 which required them to query a chatbot with a drug information question and critique the chatbot’s response. The pilot assignment guided students to critically think about the role of chatbots as a tool to obtain drug information with regards to accuracy, detail, and usability. The de-identified assignments were analyzed in an inductive process to categorize evaluations and identify emerging themes.

Results: A total of 138 assignments were collected and analyzed. Identified themes related to accuracy included alignment with current guidelines and lack of references to literature. In relation to detail, students commented on the inclusion of extraneous information and the absence of important information. In terms of usability, students identified friendly tone, easy to understand terms, and lack of relevancy as characteristics of the chatbot responses. Students who submitted simple and short questions were more likely to exhibit positive views of accuracy and usability, while students who submitted complex questions commented negatively on detail and usability. The evaluations revealed possible solutions to improving the chatbot’s response, including formal training for the students to learn how to better input a question.

Conclusions: Students demonstrated an understanding that chatbots have shortcomings in accuracy, detail, and usability, but they also expressed an interest in learning more about this technology. Improving the pilot assignment to include more instruction on how and when to use chatbots can support appropriate use of these rapidly developing AI tools.