(216) Academic Conference Posters: Maintaining Accuracy during Poster Quality Scoring

Objective : Research poster presentations are a mainstay at academic conferences. Quality of posters have varied, resulting in variations in applicability and utility of research conclusions to the academy. A literature search uncovered 11 rubrics. We compared two notable analytic rubrics (AR1, AR2) to a newer mixed-approach rubric (MAR).

Methods: Sixty randomly-selected research posters were downloaded from the 2023 AACP Annual Meeting repository. Two experienced pharmacy educators independently scored posters using the MAR, AR1, & AR2 rubrics. (Raters were not trained on any rubrics nor discussed rubric scoring prior to use.) For each rubric, traditional inter-rater reliability (by intraclass correlation), and advanced psychometric Many-Facets Rasch Measurement were used. Time-to-score was also noted.

Results: Two-way random intraclass correlations for MAR= .35(absolute-agreement) & .36(consistency), for AR1= .35(absolute-agreement) & .57(consistency), and for AR2= .37(absolute-agreement) & .57(consistency). Using Rasch, the MAR’s rating-scale functioned well, while AR1 had a misfunctioning item rating-scale, & AR2 had 4 misfunctioning item rating-scales. Rasch reliabilities were MAR=.71, AR1=.56, & AR2=.72; poster separations were MAR=2.4, AR1=1.8, & AR2=2.5. For time-to-score, AR1 & AR2 averaged 4-5minutes per poster, whereas the MAR was quicker and averaged 1-2minutes per poster.

Conclusions: Traditional inter-rater reliability indices results were equivocal; absolute-agreement appeared similar for all, though consistency favored AR1 & AR2. However, the rating-scale, foundational to any well-functioning instrument, appeared best with the MAR; its reliability and ability to distinctly separate posters into high/low quality appeared helpful (compared to the AR1 & AR2). Additionally, the MAR took considerably less time to complete than AR1 or AR2. Thus, some revisions may help improve the MAR further. A reliable rubric could aid with improving posters to communicate important research and advance the scholarship of teaching and learning.