(358) From Image to Instruction: Evaluating ChatGPT's Ability to Guide Medication Use

Objective : To evaluate the accuracy and safety of ChatGPT4-generated medication instructions from self-taken drug images compared to manufacturer patient package inserts.

Methods: Twelve medications were photographed in three ways, including by themselves, with packaging, or packaging alone. Images were uploaded to ChatGPT4 (Version Jan 2022) in September-October 2023 to generate instructions with the prompt “How do I take this medication.” Responses were collected, and the text was prepared to be compared with the PI from DailyMed. Text similarity was tested based on Cosine Similarity under Count Vectorization and TF-IDF Vectorization providing a similarity percentage to either the medication instruction for use (IFU) or MedGuide. The ChatGPT-generated medication instructions were clinically evaluated for accuracy based on meeting acceptable patient instructions.

Results: ChatGPT identified all 12 medications and produced patient text instructions on administration. Between the two similarity comparison methods, Count Vectorizer demonstrated a better similarity performance than TF-IDF with an average difference of 8.75%. The average text similarity of ChatGPT instructions to IFU and MedGuides was 76.24%. Clinical review scores were lower, with a 59.75% match. ChatGPT was found to provide more patient-friendly instructions than the PI. Medications with more complicated routes of administration (e.g. injectables) or steps were found to have lower Count Vectorization match rates and clinical review scores. Medications that were administered orally or with fewer steps to use were found to have higher match scores.

Conclusions: ChatGPT has the potential to generate step-by-step administration instructions and provides more patient-friendly language than the standard IFU or MedGuide. However, its accuracy varies with medication administration complexities and discrepancies may arise from its diverse training resources, leading to lower similarity scores on complex instructions.