(165) Exploring the Use of ChatGPT to Analyze Student Course Evaluation Comments

Objective : While course evaluations are a common tool used across higher education, the process of identifying useful information is time consuming. ChatGPT has the potential to reduce workload in higher education. The purpose of this study is to explore the use of ChatGPT in analyzing course evaluation comments, including time required to generate themes and determine the level of agreement.

Methods: Course instructors analyzed open-ended student course evaluation comments. Five prompts were provided to guide the coding process. Instructors were asked to note the time required and how they felt during their analysis. Student comments were also analyzed through two Open-AI ChatGPT user accounts. Thematic analysis was used to analyze the themes. Percent agreement between the instructor and ChatGPT themes were calculated for each prompt, along with an overall agreement statistic.

Results: There was high agreement between the instructor and ChatGPT results. The highest agreement was for course-related topics (range .71-.82) and lowest was for course weaknesses (range .53-.81). For almost all prompts, the two ChatGPT accounts demonstrated higher agreement with one another than with the instructors. On average, instructors took 27.50 ± 15.00 minutes to analyze their data (range 20-50). ChatGPT took 10.50 ± 1.00 minutes (range 10-12) and 12.50 ± 2.89 minutes (range 10-15) to analyze the data. Instructors reported feeling anxiety prior to the process, satisfaction during the process, and frustration related to findings.

Conclusions: This study offers valuable insights into the potential of ChatGPT as a tool for analyzing open-ended student course evaluation comments in health professions education. However, it is crucial to ensure ChatGPT is used as a tool to assist with the analysis and to avoid relying solely on its outputs for conclusions.