(228) Generalizing AACP Poster Quality: Rater Experience Matters

Objective : Research posters are a mainstay within academic conferences, though some posters can be of questionable quality. Previously, a mixed-approach rubric was created to quickly score poster quality. The objective herein was to evaluate consistency of rubric scores when used by multiple pharmacy education experts.

Methods: Sixty research posters were randomly selected from an online repository of the 2023 AACP Annual Meeting. Using a previously-created rubric and no training, four experienced pharmacy education faculty each scored all 60 posters. SPSS was used for descriptive statistic.; Generalizability Theory was used for consistency/reliability (three sources: poster, raters, scores; with 20posters per source). The Many-Facets Rasch Model was used for describing rating-scale and rater severity.

Results: Raters’ experience with, posters and infographics varied. Raters had the same median scores, different means, and everyone used the entire 1-4 scale. Paired correlations were low to moderate among raters. The Intraclass-correlation was .32 for 4-raters (all-experience), while .43 for 3-raters (more-experienced), and .53 for 2-raters (most-experienced) (all p<.001). Three separate G-studies showed a similar trend for reliability (consistency); the 2-rater reliability was 0.5 (all-experience), 0.6 (more-experienced), and 0.7 (most-experienced). The rating-scale functioned well with distinct use of all rating-scale categories, while lenient and strict raters were modelled (the most experienced raters had similar Rasch-measures, while others were substantially more lenient and more stringent).

Conclusions: Experience of raters mattered for scoring reliability/consistency. Knowing this, rater experience should be characterized before evaluating poster quality, with rater training efforts focused on potential raters with lesser experience. While some rater training might be helpful, this instrument appears positive for judges to rate poster quality, as well as a “checklist” as conference-attendees prepare their posters for conference presentation.