(279) Mustering Machines’ Meticulous Mastery: A Machine-Learning Model for Classifying Pharmacy Practice Publications into Research Domains

Objective : To establish an objective, machine-learning classifier for faculty publications into research domains (Clinical, Social & Administrative, Education, and Translational), to address the heterogeneity in pharmacy practice faculty research publications.

Methods: We finetuned a series of Bidirectional Encoders Representations from Transformers (BERT) large language models (LLMs) pre-trained on biomedical corpora (PubMed ± Clinical Notes (MIMIC III)) in a Python environment. Our training dataset was 1,000 random samples of abstracts from pharmacy practice faculties’ 2018-2021 publications that we manually labeled. Our final model was further validated using 80 random samples of abstracts from 2023 publications, labeled with ≥80 consensus by all authors. Using 4,990 abstracts from 2018-2021 faculty publications that the model was naïve to, we demonstrated a use case of the classifier by testing the null hypothesis that the research domain distribution of faculty publications is independent of the pandemic (2018-2019 vs. 2020-2021), using the Chi-Square test of independence.

Results: The training dataset included 51.5% Clinical, 26.4% Social & Administrative, 16.9% Education, and 5.2% Translational research documents. The selected model had a 5-fold stratified cross-validation metrics including 95.3±0.7%, 90.2±2.5%, 88.2±4.5%, 88.7±2.3%, for accuracy, precision, recall, and F1 scores, respectively, and correctly classified 96.3% of the 2023 publications. The research domain distribution of articles was independent
of the pandemic period despite some fairly notable shifts (Clinical: 55.1%-52.1%, and Translational: 4.0%-4.9%; χ2 (3, 4990) = 5.92, p = .11).

Conclusions: The state-of-the-art performance of this automated classifier will foster consensus and consistency in pharmacy practice, which are key goals of the Granada Statements. It will allow for advanced, hitherto impossible, granular bibliometric analysis and could be an objective tool for setting and evaluating research and scholarship benchmarks for faculty.