A retrospective interrupted time series analysis of alcohol withdrawal protocol changes on patient outcomes and hospital metrics

The Clinical Institute Withdrawal Assessment for Alcohol, revised (CIWA-Ar, or CIWA) is a clinical evaluation protocol for alcohol withdrawal patients, in use since the 1980s.1 The CIWA is comprehensive, well-established, and validated in various patient populations.

A Retrospective Interrupted Time Series Analysis of Alcohol Withdrawal Protocol Changes on Patient Outcomes and Hospital Metrics MSTAT Biostatistics Final Project Defense Bryce Johnson Committee Members: Dr. Christy Porucznik, Professor Greg Stoddard, Dr. Andrew Wilson November 22, 2024 Introduction The Clinical Institute Withdrawal Assessment for Alcohol, revised (CIWA-Ar, or CIWA) is a clinical evaluation protocol for alcohol withdrawal patients, in use since the 1980s.1 The CIWA is comprehensive, well-established, and validated in various patient populations. Alcohol withdrawal evaluations occur frequently throughout a patient’s visit, causing the lengthy, more time-consuming CIWA to be burdensome. This burden impacts hospital staff and resources, including valuable staff time and cognitive load. In 2017, Dr. Rastegar et al. developed a streamlined alcohol withdrawal evaluation criteria called the Brief Alcohol Withdrawal Scale (BAWS) to remedy the limitations of the CIWA.2 The BAWS maximizes evaluation criteria value while minimizing the toll on hospital staff and resources. At the University of Utah Hospital, a medical research team replaced the CIWA for the BAWS in April 2022 for their alcohol withdrawal patients. The present study analyzed how the updated BAWS evaluation criteria impacted patient outcomes and hospital metrics via an Interrupted Time Series (ITS) Analysis. Methods Data Source The data came from the University of Utah Enterprise Data Warehouse. The initial patient data pull used ICD codes related to alcohol withdrawal provided by the medical research team, with dates ranging from April 1, 2021, to April 1, 2023. Exclusion criteria included patients without a CIWA or BAWS score, patients who did not have a medicine team assigned within 24 hours, patients who were admitted to the ICU first, and any other patients who were flagged as emergency or bedded outpatient patient types. This resulted in 1341 total unique visits and 883 total unique patients. The preintervention group had 665 unique Figure 1. Alcohol Withdrawal Patient Data Lineage visits and 480 unique patients. The post-intervention group had 676 unique visits and 486 unique patients (see Figure 1). Variables of Interest The primary outcomes of interest were continuous variables, including the amount of benzodiazepines administered, hospital length of stay (in days), and patient cost (in inflationadjusted 2021 dollars). The secondary outcomes of interest included binary categorical variables to assess whether the intervention was associated with the following outcomes: mortality, readmission, discharge to an intensive care unit (ICU), patient left against medical advice, and discharge prescriptions Naltrexone, Acamprosate, and Disulfiram. The intervention is a binary categorical variable, whether the patient received the CIWA (0 for pre-intervention) or the BAWS (1 for post-intervention). Covariates and adjustment variables included age, Charlson Comorbidity Index (CCI), a median BAWS score for the patient’s visit, gender, whether the patient’s visit was ever transferred to the ICU, and whether phenobarbital was administered. Since a patient receives alcohol withdrawal evaluations every few hours during one hospital visit, a single median BAWS score is obtained for that visit. A combination of alcohol withdrawal pharmacologic interventions created the primary outcome amount of benzodiazepines administered by converting them to a shared unit of lorazepam equivalent units. The medical research team provided these conversions. The combined pharmacologic interventions included chlordiazepoxide (10mg to 1mg lorazepam), diazepam (6mg to 1mg lorazepam), lorazepam, phenobarbital (15mg to 1 mg lorazepam), and phenobarbital sodium (15mg to 1 mg lorazepam). There is a risk for collinearity between the amount of benzodiazepines administered and the binary categorical variable, phenobarbital administered. Using the vif() function from the R package “car”, the Variance Inflation Factor output did not indicate concerning levels of collinearity (VIF ~<1.5). Statistical Model An Interrupted Time Series (ITS) analysis is a statistical model used to evaluate the impact of an intervention or event on a specific outcome over time. ITS is particularly useful in medical research for assessing changes before and after implementing a protocol change. ITS is arguably one of the strongest quasi-experimental (i.e., non-randomized) study designs, as it helps infer causation without a randomized control group, which may be missing due to feasibility or ethical reasons.3 This research question is particularly suited for an Interrupted Time Series – Regression Discontinuity design, as this design detects an immediate and distinct effect of an intervention at a specific point. The alternative method, Interrupted Time Series – Longitudinal, is inappropriate for this research question as that design requires repeated measures data. ITS designs come with several limitations, such as their retrospective design and, most especially, their susceptibility to confounding events. Other events or policies occurring around or throughout the same time as the intervention or study period can complicate the interpretation. Fortunately, some limitations, such as autocorrelation and seasonality, can be modeled and adjusted. This version of the Interrupted Time Series equation comes from the work of Linden et al. for a single-group analysis.4,5 Yt is the aggregated outcome variable measured at each equally spaced time point, t. Tt is the time since the start of the study, in the same time units as Y t. Xt is the binary indicator for intervention: 0 for Pre-Intervention and 1 for PostFigure 2. Interrupted Time Series Equation Intervention. XtTt is the interaction term between the binary indicator for intervention and the time since the study started. 0 is the intercept or starting level of the outcome variable. 1 is the slope or trajectory of the outcome variable until the introduction of the intervention. 2 is the change in the outcome level that occurs immediately following the intervention's introduction, also called the “Step Change.” 3 is the difference between the preintervention and post-intervention slopes of the outcome. Thus, a significant p-value in 2 indicates an immediate intervention effect, and a significant p-value in 3 indicates a treatment effect over time. Additional covariates and coefficients can be added to the model for relevant context and adjustment factors. Details on the error term can be found in Linden’s original paper. Figure 3. Generic Interrupted Time Series Plot Figure 3 shows a typical representation of an Interrupted Time Series plot. On the x-axis is time Tt. On the y-axis is the continuous outcome Yt. The vertical red dashed line in the center indicates when the intervention occurred in time. The pre-intervention line is blue, and the post-intervention line is orange. The counterfactual line is anticipated to have happened, assuming the intervention had not occurred and the pre-intervention trends continued. The counterfactual line is the grey dashed line. Intercept 0 is the outcome value at both Time and Intervention = 0. 1 is the preintervention slope. 2 is the intervention's step change or immediate effect on the outcome. 3 is the difference in the slopes or the impact of the intervention on the outcome over time. Results Table 1 Table 1 shows the patient characteristics split by pre-intervention and post-intervention visits. The variables include Age, Gender, Charlson Comorbidity Index (CCI), Patient Type, median BAWS, and Payor. The three variables that are significant, or are very close to significant, are CCI, Patient Type, and median BAWS. The median BAWS shows a significant p-value, though this is likely due to confounding present in the crosswalk process converting all preintervention CIWA values to BAWS. The differences in patient type are not large enough to be concerning. CCI, however, shows identical median values and similar interquartile ranges of 2 at a significant p-value of 0.007. Figure 4 shows how the pre-intervention group tends to drag the Table 1. Patient Characteristics Figure 4. Charlson Comorbidity Index Plots, Statistics CCI higher than the post-intervention group via boxplot, density plot, and summary statistics. Table 2 Table 2 shows a summary of unadjusted patient outcomes with no time effect. The amount of benzodiazepines on a visit is statistically significant, with a lower median benzodiazepine value in the post-intervention. Phenobarbital shows a statistical difference in an increase in phenobarbital in the postintervention group. The variable ICU during stay shows a statistical difference with increased ICU admissions in the post-intervention group. Length of stay was not shown to be statistically different. The three discharge prescription drugs were shown to be statistically different. The Disulfiram value, in particular, shows a pre-intervention value of only two visits receiving that prescription. Finally, the groups had a nearly significant difference in discharge disposition. Table 2 continues with patient financial outcomes. All dollar amounts are replaced with scaled percent differences to maintain data confidentiality. The results indicate only a statistically significant increase in Lab costs and a significant decrease in expenses in the Other category. All dollar values are inflation-adjusted to 2021 dollars. Autocorrelation One critical assumption check in an ITS study is autocorrelation. For data spanning two years, the time unit of weeks balances the need for granularity of the visits while also demonstrating the outcome result patterns over time. R's acf() function showed no Table 2. Patient Outcomes problematic patterns for all continuous timerelated outcomes. Figure 5 illustrates the Autocorrelation Factor (ACF) for weekly average benzodiazepines on the y-axis and Lag (in weeks) on the x-axis. A problematic ACF exists if a gradually ascending or descending pattern passes the blue dashed line. There is no such concerning pattern here. For confirmation, a Durbin-Watson test was used to solidify the result of no evidence of autocorrelation (p-value = 0.4844). Choice of Linear Model Many linear modeling Figure 5. Autocorrelation Factor of weekly average approaches exist. A bootstrapped benzodiazepines via R function acf() Cullen and Frey graph was used initially to find the best-fitting linear models to review the appropriate distributions for the outcome (500 bootstraps). From there, the most suitable model was picked based on R plot() function readouts, including residual, normality, and leverage of data points plots. The best-fitting model was chosen based on each outcome's most homoscedastic, most normal, least leveraged, and lowest AIC models. For the outcome amount of benzodiazepines, this was the linear model lm(). The best model for outcomes length of stay and relative cost was a gamma glm() with an identity link. Amount of Benzodiazepines Figure 6 shows the results of the bestfitting linear model lm() for the outcome of weekly average benzodiazepines, predicted by the interrupted time series components and adjustment covariates. ic_weeks_since_start is an intervention-centered time variable to maximize interpretability. The beta coefficients represent the average change in the outcome (weekly average benzodiazepines) for a one-unit increase in each covariate, holding other variables constant. The step change coefficient, Intervention, is not statistically significant. Nor is there a difference in slopes, the interaction term between ic_weeks_since_start and Intervention. The statistically significant betas are a positive average median BAWS coefficient and a positive Figure 6. Amount of Benzodiazepines lm() Results percent phenobarbital coefficient. These results coincide with expected values, as patient visits with a higher BAWS score ought to correlate with higher amounts of benzodiazepines administered. Higher amounts of benzodiazepines are also likely in patients who receive phenobarbital. Figure 7 illustrates these results in an interrupted time series plot. On the y-axis, there is a weekly average of benzodiazepines in lorazepam equivalent Figure 7. Weekly Average Amount of Benzodiazepines Interrupted Time Series Plot units. On the x-axis, there is time in weeks. In blue are the pre-intervention lines with their associated 95% confidence interval bands and a predicted (covariate-adjusted) best-fit straight dashed line. In orange are the post-intervention lines with their 95% confidence interval bands and the predicted best-fit straight dashed line. The counterfactual is visualized in the light grey dashed line for comparison against the post-intervention weeks. The p-values for both the step change and the slope change are insignificant. This indicates that the intervention did not impact the weekly amount of benzodiazepines administered either immediately or over time. Length of Stay The best fitting model for the length of stay outcome was the gamma glm() with identity link. The results in Figure 8 show that the step change and slope differences were also not shown to be statistically significant. The average age and weekly percentage of patients who went to the ICU during their stay had a statistically significant relationship with length of stay. It holds to reason that those who are older and those who require ICU admission may need a longer stay than their counterparts on average. Figure 9 shows these results in an interrupted time series plot. The pvalues indicate that the intervention did not have a significant relationship with a weekly average length of stay, either via an immediate effect or time effect via slope differences. Figure 8. Length of Stay glm() Results Figure 9. Weekly Average Length of Stay (Days) Interrupted Time Series Plot Relative Cost The best fitting model for the relative cost outcome was the gamma glm() with identity link. All dollars were inflation-adjusted to 2021 dollars and then were replaced with scaled percent differences to maintain data confidentiality. The results in Figure 10 show that the step change was not shown to be statistically significant. However, the difference in slope terms was statistically significant (p = 0.014), albeit with a minuscule beta coefficient (-0.003). The weekly percentage of patients who went to the ICU during their stay had a statistically significant relationship with relative cost. This ICU variable result coincides with the expectation that those who go to the ICU pay Figure 10. Relative Cost glm() Results more on average than those who do not. Figure 11 shows these results in an interrupted time series plot. The significant difference in slopes is steeper in the post-intervention group. Some extreme relative cost values exist shortly after the red dashed intervention line. The medical research team was concerned about this result potentially dragging up the post-intervention line steepness and requested a sensitivity analysis. Figure 11. Weekly Average Relative Cost (%) Interrupted Time Series Plot Relative Cost – Sensitivity Analysis A sensitivity analysis is a method that slightly adjusts the original data to see if those adjustments change the original model’s results. When performing any sensitivity analysis, researchers must carefully interpret results as they are inherently biased. In this case, the study’s three most extreme relative cost value visits were dropped, and the results were analyzed. Selection of data points based on their outcome value biases the results. Figure 12 shows a reduced model where the three most extreme relative cost value visits were dropped. The initially statistically significant percent_icu_during_stay remained so, while the slope difference became statistically insignificant. Figure 13 visualizes the data without those three most extreme cost visit values. The slope difference is less steep or dramatic in the post-intervention than in the full model. Figure 12. Reduced Relative Cost glm() Results (Sensitivity Analysis) Figure 13. Reduced Weekly Avg. Relative Cost (%) Interrupted Time Series Plot (Sensitivity Analysis) Logistic Regression Results For the remaining results, logistic regression modeling approaches were used. No time effect was analyzed using this method nor for any logistic regression in this study. Figure 14 shows the results of associations with mortality and associated covariates, including a binary indicator for Intervention. The results are given in odds ratios for the beta coefficient and 95% confidence intervals. The comparison group for the odds ratio result is pre-intervention females with no phenobarbital. Intervention was not significantly associated with the mortality binary variable. The only statistically significant relationship with mortality is the median BAWS variable (p < 0.001), with an odds ratio coefficient of 1.96. This means that a oneunit increase in median BAWS is associated with nearly double the odds of mortality. For context, out of the 1341 total patient visits analyzed, nine were reported as having suffered mortality. Their median BAWS scores were higher than the average visit. Note that the variable “ICU during stay” was Figure 14. Mortality Logistic Regression Results dropped for the logistic regression due to its high odds ratio and statistical insignificance (p-value > 0.99). The remaining logistic regression result figures are included in the appendix, and statistically significant results are reported here. For the readmitted binary variable, age was significantly associated (p < 0.001) with an odds ratio coefficient of 0.98. CCI was also significantly associated (p < 0.001) with an odds ratio coefficient of 1.14. Lastly, median BAWS was significantly associated (p < 0.001) with an odds ratio coefficient of 1.41. No other variables were significantly associated. The post-intervention group was significantly associated with the ICU admission binary variable (p = 0.023) with an odds ratio coefficient of 1.80. The benzodiazepines variable was also significant (p = 0.002) with an odds ratio coefficient of 1.01. Median BAWS was significant (p = 0.007) with an odds ratio coefficient of 1.31. Phenobarbital was significant (p < 0.001) with an odds ratio coefficient of 4.06. No other variables were significantly associated. This high odds ratio of ICU-admitted patients receiving phenobarbital was worth exploring as an interaction term to compare Intervention and phenobarbital. With the interaction term between Intervention and phenobarbital included, the interaction is not statistically significant (p = 0.45). Further, the postintervention group is no longer statistically significant (p = 0.21). Median BAWS remains statistically significant (p = 0.008) with an odds ratio coefficient of 1.31. Phenobarbital remains statistically significant (p = 0.016) with an odds ratio coefficient of 3.11. The benzodiazepines variable also remains statistically significant (p = 0.002) with an odds ratio coefficient of 1.01. No other variables were significantly associated. Intervention is not significantly associated with the leaving against medical advice (LAMA) binary variable. Age is significantly associated (p = 0.006) with an odds ratio coefficient of 0.97. Median BAWS is also significantly associated (p < 0.001) with an odds ratio coefficient of 1.78. No other variables were significantly associated. For the discharge drug binary variables, the intervention group was significantly associated with each of them. For naltrexone, the post-intervention group is significantly associated with an increase in prescription (p = 0.024) and an odds ratio coefficient of 1.35. CCI is also significantly associated (p = 0.002) with an odds ratio coefficient of 0.88. And the benzodiazepines variable is significantly associated (p = 0.001) with an odds ratio coefficient of 1.01. No other variables were significantly associated with naltrexone. For acamprosate, the post-intervention group is significantly associated with an increase in prescription (p = 0.037) with an odds ratio coefficient of 1.48. No other variables were significantly associated with acamprosate. For disulfiram, the post-intervention group was significantly associated (p = 0.003) with an odds ratio coefficient of 9.32. This odds ratio is exceptionally high because only two out of 665 pre-intervention patient visits received disulfiram, whereas 17 out of 676 post-intervention patient visits received disulfiram. No other variables were significantly associated with disulfiram. Discussion Intervention-associated Results Across the primary outcomes of interest in the amount of benzodiazepines administered, length of stay, and relative cost, only relative cost was statistically significantly associated with a difference in intervention-induced slope change (p-value = 0.0136, beta = -0.003). After a sensitivity analysis dropping the three most extreme cost values, however, that slope change was shown to be statistically insignificant (p-value = 0.0679). Due to the biased nature of this sensitivity analysis selection based on the relative cost outcome, this result should be interpreted carefully. For the ICU admission binary variable, the post-intervention group was significantly associated (p = 0.023) with an odds ratio coefficient of 1.80, and phenobarbital was significant (p < 0.001) with an odds ratio coefficient of 4.06. This means that the intervention increased the odds of ICU admission. This was worth looking into the interaction term between intervention and phenobarbital. With the interaction term between intervention and phenobarbital included, the interaction is not statistically significant (p = 0.45). The post-intervention group was no longer statistically significant (p = 0.21), and phenobarbital remained statistically significant (p = 0.016) with an odds ratio coefficient of 3.11. This means that both models show an increase in ICU admission for those patient visits receiving phenobarbital. Finally, all discharge prescriptions were statistically significantly associated with the intervention, showing higher odds of prescription in the post-intervention group. There may have been a change in the discharge prescription protocol sometime during the study period. Limitations The limitations of this study include: First, this study uses a retrospective cohort design, which may introduce biases related to the selection of subjects and data availability. Retrospective analyses also depend on the recorded data's quality and completeness. Second, other unmeasured confounders could influence the outcomes despite adjusting for several covariates. The study period also spans over a year before and after the intervention, during which other changes in clinical practice or hospital policies affecting the intervention or covariates could have occurred (e.g., updated COVID-19 protocols). Third, the variability in BAWS scores, which replaced CIWA scores, could introduce inconsistencies in the measurement of alcohol withdrawal severity. Fourth, investigating multiple outcomes in the same study increases the risk of type 1 errors (false positives). Although adjustments for multiple comparisons were made, doing so may reduce statistical power and increase the risk of type 2 errors (false negatives). Fifth, the method of inflation adjustment of all reported costs to 2021 dollars might not fully account for all economic factors influencing healthcare costs during the study period, potentially affecting the interpretation of cost-related outcomes. Sixth, the sensitivity analysis approach may not reflect the results of different model specifications and assumptions (e.g., different links for the gamma GLM or other model distributions), so the sensitivity analysis results may not cover all potential model variations. Lastly, the findings from the University of Utah hospitals and clinics may not be generalizable to other institutions with different patient populations, protocols, and healthcare systems. Future Directions The future directions for this research include: First, a comparison to a parallel university hospital, even without an intervention, may be appropriate to establish even more robust results than observed here. Second, a cluster and factor analysis could be helpful, where using a combination of patient characteristics and outcomes may provide different results. Finally, targeted maximum likelihood estimation (TMLE) is a causal methods approach that lends itself well to observational data and may help determine the causal relationship between intervention and the outcomes. Combining several advanced techniques may indicate differing results from those reported here. Takeaway The results shown here indicate that the more cumbersome CIWA evaluation criteria can be replaced by the more streamlined BAWS with no significant drawbacks in the variables measured in this study. The BAWS generally improves the evaluation experience for hospital staff by requiring less time and cognitive load while maintaining evaluation quality. Conclusion This retrospective interrupted time series analysis aimed to assess the impact of transitioning from the CIWA to the BAWS evaluation criteria on patient outcomes and hospital metrics at the University of Utah hospital. While the results suggest that the BAWS implementation did not significantly alter primary outcomes such as benzodiazepine administration, hospital length of stay, or patient costs, it demonstrated associations with specific secondary outcomes, including increased odds of ICU admissions in phenobarbital-administered cases and higher rates of discharge prescriptions. These findings reflect the nuanced effects of the protocol change, particularly in areas like pharmacological management and discharge practices. References 1. 2. 3. 4. 5. SULLIVAN JT, SYKORA K, SCHNEIDERMAN J, NARANJO CA, SELLERS EM. Assessment of Alcohol Withdrawal: the revised clinical institute withdrawal assessment for alcohol scale (CIWA‐Ar). Br J Addict. 1989;84(11). doi:10.1111/j.13600443.1989.tb00737.x Rastegar DA, Applewhite D, Alvanzo AAH, Welsh C, Niessen T, Chen ES. Development and implementation of an alcohol withdrawal protocol using a 5-item scale, the Brief Alcohol Withdrawal Scale (BAWS). Subst Abus. 2017;38(4). doi:10.1080/08897077.2017.1354119 Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13(6 SUPPL.). doi:10.1016/j.acap.2013.08.002 Linden A, Adams JL. Applying a propensity score-based weighting model to interrupted time series data: Improving causal inference in programme evaluation. J Eval Clin Pract. 2011;17(6). doi:10.1111/j.1365-2753.2010.01504.x Linden A. Conducting interrupted time-series analysis for single- and multiple-group comparisons. Stata Journal. 2015;15(2). doi:10.1177/1536867x1501500208 Appendix: glm(Readmitted_binary ~ Intervention + Age + Gender + CCI + MEDIAN_BAWS + `ICU During Stay` + Phenobarbital + `Benzos On Visit`, data = df, family = binomial) glm(ICU_binary ~ Intervention + Age + Gender + CCI + MEDIAN_BAWS + Phenobarbital + `Benzos On Visit`, data = df, family = binomial) glm(ICU_binary ~ Intervention * Phenobarbital + Age + Gender + CCI + MEDIAN_BAWS + `Benzos On Visit`, data = df, family = binomial) glm(LAMA_binary ~ Intervention + Age + Gender + CCI + MEDIAN_BAWS + Phenobarbital + `Benzos On Visit`, data = df, family = binomial) glm(discharge_drug ~ Intervention + Age + Gender + CCI + MEDIAN_BAWS + Phenobarbital + `Benzos On Visit`, data = df, family = binomial) Naltrexone Acamprosate Disulfiram