(273) AI Grading versus Traditional Grading for Pharmacy Writing Assignments

Objective : Evaluate the performance of automated intelligence (AI) in grading pharmacy student writing assignments in a drug information course.

Methods: A writing assignment from a P1 drug information course taught in Fall 2023 was selected for the study. The maximum score possible was 30 points. ChatGPT4.0 from OpenAI was given the assignment prompt and grading rubric and asked to score student papers. Each paper was evaluated by ChatGPT five times to check for consistency in the AI grades. A sixth ChatGPT run was conducted to train ChatGPT using 5 faculty-graded assignments. Two researchers, separate from the course instructor, managed the inputs to ChatGPT. Absolute agreement intraclass correlation coefficients (ICC) were calculated between each ChatGPT grading run and the faculty grader.

Results: All papers from the assignment were included (N=32). The range of scores from the instructor was 25-30 points (mean 27.53 ± 1.65). Scores from ChatGPT ranged from 19-30 points, with the means from the 5 main runs ranging from 27.56-29.65 points (overall mean 28.62 ± 1.38). The absolute agreement was poor between the instructor and GPT on total scores, with ICCs ranging from 0.100 to 0.254. Training ChatGPT before grading the remaining papers (n=27) did not improve the absolute agreement with the instructor (ICC=0.157).

Conclusions: ChatGPT4.0 did not consistently agree with the instructor when grading a P1 writing assignment. While the AI program did not agree with the instructor for this particular assignment, AI’s performance may differ on other assignments or topics. Additional work is needed to explore and monitor AI’s performance in coursework grading.