(461) Point-in-Time Assessment: Incorporating APPE Data to Measure Program Outcomes and Make Rapid-Cycle Program Improvements

Objective : To design and implement a 'Point-in-Time' assessment approach to retrieving, analyzing, summarizing, and reporting APPE outcomes to inform more timely program improvements.

Methods: The APPE performance assessment items were mapped to program outcomes, with data analyzed on a block-by-block basis and reported to key stakeholders at critical times during the academic year to support data-informed decision making. Through the development of heat maps and evaluation of qualitative comments, individual and aggregate student and preceptor performance was reviewed using a ‘point-in-time’ approach in addition to further aggregate analysis at the end of the APPE cycle.

Results: First, we were able to identify and provide directed support to individual students who were not performing at expected standards both early on and throughout their APPEs, as well as identify preceptors who might need additional training regarding the student performance assessment. Second, the new process allowed us to gain insight into elements of the new EPA-based evaluation instrument that required revision to provide greater clarity; we were able to implement changes prior to the start of the next APPE cycle. Third, this process facilitated identification of a program outcome (critical thinking) where performance was slightly below benchmark and was supported with data from other courses. Early identification of this program outcome led to focused discussions around the specific outcome, particularly at our end-of-year curriculum summit.

Conclusions: The new process allowed for more timely and effective use of APPE data to support learners and drive program improvements.