Policy Library Quality Improvement: Redundancy Analysis using BERTopic and Llama 2 Artificial Intelligence Clustering

Policies and procedures are key components of healthcare system governance that inform the safety and quality of patient care. Healthcare organizations develop digital policy libraries to organize these documents, but without regular maintenance, redundancies may occur that reduce the library's accuracy and efficiency. Manual categorizing of similar policies to find redundancies for consolidation is a time- consuming process that was identified for potential quality improvement with technological solutions. Methods: A quality improvement project was developed to find an automated method to identify similar policies within the healthcare organization's policy library. The policy lifecycle of the organization was mapped to help illustrate its workflow. BERTopic, a clustering algorithm, was utilized to identify similarities with a large language model called Llama 2 for Artificial Intelligence (AI) generated topic labeling. Results: The open-source clustering method was applied first to a sub-section of the policy library to investigate the potential for identifying similarities. Out of 175 policies, 18 were flagged as similar for further investigation. The algorithm was then applied to the full policy library. The manual process of clustering the policy library took the Accreditation team a total of 225 hours with a team of three people. Clustering the policy library and flagging similar policies took a single person 2 hours of runtime for the algorithm and 25 hours for flagging similar policies, resulting in a time savings of 89%. A modified workflow using this algorithm was proposed as an addition to the maintenance phase of the policy library lifecycle. Conclusions: The use of AI algorithms powered by open-source code is a replicable and low-cost solution with significant time savings that should be studied further. The academic background of policy maintenance in a healthcare organization is not well documented in the available literature.