Impact of Quality Improvement (QI) Program on 5-Year Risk of Diabetes-Related Complications: A Simulation Study

A large number of individuals with diabetes fail to meet their glycemic guidelines (1–3). There has been underuse of process improvement and system-wide strategies to improve treatment adherence, despite increasing disease knowledge, new technologies, and advocacy by physicians and health care organizations (4,5). To improve the quality of care for all individuals with diabetes, the American Diabetes Association (ADA) initiated the Diabetes INSIDE (INspiring System Improvement with Data-Driven Excellence) program. This program was developed based on established quality improvement (QI) principles and the Standards of Medical Care in Diabetes (6); it is a structured, system-wide program of care system redesign, provider and patient training, education, and support implemented at leading health care systems across the U.S. (7).

We have successfully implemented the ADA’s Diabetes INSIDE QI program at a university hospital and safety-net health system (Tulane and Parkland Health & Hospital System [Parkland]), focused on system-wide improvement in individuals with poorly controlled type 2 diabetes mellitus (T2DM). The specific QI strategies of Diabetes INSIDE we implemented include: 1) individual provider and departmental reports on clinical outcomes, 2) patient outreach programs to improve follow-up care, 3) campaigns to raise patients’ understanding of achieving clinical goals, 4) improving population monitoring through improving electronic health record (EHR) data capture, 5) professional education and practice delivery redesign, 6) an insulin initiation and intensification program using shared medical appointments, and 7) local QI committees to design and manage the initiatives. These QI interventions targeted all adults treated for T2DM in the family medicine, internal medicine, and endocrinology departments. The overall design of the Diabetes INSIDE programs have been previously described (7).

The clinical efficacy of the QI program on reducing HbA_1c under testing, the proportion of individuals with poorly controlled HbA_1c, and mean HbA_1c level at the Tulane site were reported recently (8,9). Our analysis revealed a 15.5% relative improvement in the patient proportion with HbA_1c >9% (75 mmol/mol) following QI interventions and a 2.1% reduction of population mean HbA_1c from 7.4% (57 mmol/mol) to 7.2% (55 mmol/mol) (P < 0.05).

There are large gaps in diabetes care for a significant number of patients. Our hypothesis is that population health management strategies, such as Diabetes INSIDE, with current therapies may potentially have profound effects on improving clinical outcomes. However, previous studies using surrogates like HbA_1c to evaluate the efficacy of QI intervention can only help to provide short-term insights on the program’s efficacy. There is a substantial time lag between the improvement of HbA_1c control and the prevention of complications in a longer period. Thus, there is an urgent need for both clinicians and policymakers to appreciate the magnitude of QI-related improvements on the diabetes-related cardiovascular and microvascular outcomes to assess the true impact of the QI program. Such appreciation may encourage a more wide-spread use of such population health programs.

Advanced analytical research using observational data from the QI interventions can model predicted hard outcomes and help put into better perspective the potential benefits of the QI intervention. Simulation research has a long history of supporting diabetes-related research. It was often used to translate the short-term clinical efficacy of diabetes treatment on key biomarkers into cardiovascular and mortality benefit predictions in a longer period. There are several diabetes models on the market, such as the UK Prospective Diabetes Study (UKPDS) Outcomes Model, the Centers for Disease Control and Prevention-Research Triangle Institute (CDC-RTI) Diabetes Cost-effectiveness Model, and the IQVIA Core Diabetes Model (10). The Building, Relating, Assessing, and Validating Outcomes (BRAVO) diabetes model represents a novel tool that was recently developed (11), successfully validated, and calibrated globally (12). It is a person-level discrete-time microsimulation model, predicting the progression of diabetes based on individuals’ dynamic characteristics and treatment regimen over a lifetime. The model provides predictions on the risk of macrovascular events (i.e., myocardial infarction [MI], congestive heart failure [CHF], stroke, angina, and revascularization), microvascular events (i.e., chronic kidney disease, end-stage renal disease [ESRD], retinopathy, blindness, neuropathy, and amputation), and adverse events (e.g., hypoglycemia and diabetic ketoacidosis) over a user-specified time horizon.

The objective of this study is to simulate and evaluate the projected 5-year risk reduction in macrovascular and microvascular events, as well as mortality, due to the QI intervention. We used the BRAVO diabetes model to project the short-term clinical benefit of the QI intervention to estimate 5-year diabetes-related outcomes.

Our previous study examined the impact of the QI program on HbA_1c levels in patients with diabetes from Tulane (8). In this study, we used the EHRs extracted from both Tulane and Parkland Health & Hospital System to evaluate the short-term clinical efficacy of the QI intervention. Key biomarkers to measure clinical efficacy not only included HbA_1c, but also systolic blood pressure (SBP), LDL cholesterol (LDL-C), and BMI. Due to a lack of control group, this study used a pre-post single-arm design to evaluate the treatment effect of the QI intervention, which is a similar approach to the one we published previously to evaluate the QI intervention on the Tulane experience (8). All individuals with T2DM are the subjects of the QI intervention and thus will be included in the analysis. The baseline period was defined as 1 year before the start of the QI intervention, while the intervention period was defined as the 12 months following the start of the QI intervention. The postintervention period used to evaluate the direct impact of the QI effort on biomarkers was defined as the subsequent 6 months. Parkland and Tulane had a different starting time for the QI intervention (Fig. 1). The EHR data were extracted from multiyear records of all patients with type 2 diabetes who had visited during both the pre- and postperiods. We only included individuals with HbA_1c >8% (64 mmol/mol) at the baseline in this study because the QI intervention was aiming to improve the clinical outcomes among those with poorly controlled glycemic level. Due to the geographic location of the sites, we were able to study the impact of the intervention in lower socioeconomic and ethnic groups (African American and Hispanic) who are disproportionally affected by diabetes and its complications.

The average values of biomarkers, including HbA_1c, SBP, LDL, and BMI, were summarized at both baseline and postintervention periods. The clinical efficacy of the QI program was estimated by taking the difference between values from the postintervention period and the baseline period. A t test was then conducted correspondingly to examine if the clinical efficacy is statistically significant from zero.

The simulation was conducted using the BRAVO diabetes model. The baseline characteristics for each person included in the study were extracted directly from the EHRs. All baseline variables summarized in Table 1 are required for the BRAVO model’s input, which were extracted based on EHR fields for the physical, medical, laboratory, and ICD codes for each individual patient. Deriving comorbidity information from EHRs relies on clinician accuracy and completeness of records as well as the ability to extract the corresponding information. The included diagnoses of comorbidities were based on ICD diagnoses and problem lists that may be subjective, overlooked for more urgent concerns, referred to other specialists, or lack specific interventions. One of the critical variables needed for the simulation model is the duration of diabetes. Unfortunately, it is challenging to estimate this variable from the EHRs. We have previously developed a predictive model using a random forest algorithm to predict the duration of diabetes, based on variables routinely collected in EMRs: age, current HbA_1c, retinopathy history, number of oral antidiabetic drugs, number of insulin classes, and use of sulfonylureas (13). We used this algorithm to assign a diabetes duration to each person in the simulation.

Based on patients’ characteristics and the trajectory of biomarkers, two simulation scenarios were conducted to evaluate the clinical benefits of the QI program at 5 years. In the first, baseline biomarkers (i.e., HbA_1c, SBP, LDL-C, and BMI) remained constant over the course of 5 years, serving as the control group. In the second simulation, the baseline biomarkers were changed as observed in the EHRs and remained constant over time, following evidence from the previous studies (14–16). For each simulation scenario, five macrovascular events (i.e., MI, CHF, stroke, angina, and revascularization), three microvascular events (i.e., ESRD, blindness, and neuropathy), as well as mortality rates were estimated for a 5-year window. We then calculated the relative risks (RRs) for the QI program by comparing the cumulative incidence of each outcome in the 5-year study period between the two scenarios. We have also input the upper and lower bounds of the CI of the clinical efficacy to populate the simulation, in order to estimate the probability CIs of the RR for the QI program. The persistence of the QI program relied on how the QI program was implemented and whether the physicians follow the QI program’s protocol. Thus, as a conservative estimate, we have also conducted a sensitivity analysis assuming treatment effect diminishes at a rate of 30% per year. We have summarized the results from the sensitivity analysis in Supplementary Appendix 1.

The simulation was conducted at an individual level, and the model simulated the projected outcome event probabilities 100,000 times for each patient before averaging individual outcome estimates to calculate the population estimate run 100,000 times for each person to get converged estimates.

Records of 2,429 individuals with HbA_1c >8% (64 mmol/mol) at baseline were extracted from the EHRs. The average baseline characteristics of the identified individuals were summarized in Table 1. We’ve summarized the baseline characteristics among the overall population (i.e., HbA_1c >8% [64 mmol/mol]) and two subgroups with HbA_1c 8.0% (64 mmol/mol) to 9.5% (80 mmol/mol) and >9.5%. The average age of the identified population is 56.9 years old, with 10 years of diabetes duration, 64% female, and 57% Hispanics. The average baseline HbA_1c was 9.7%. Baseline SBP, LDL-C, and BMI are 134.9 mmHg, 100.1 mg/dL, and 33.3 kg/m², respectively.

We summarized the clinical efficacy of the QI program on key biomarkers in Table 2. The QI program was associated with reductions in HbA_1c (−0.84%) and LDL-C (−5.94 mg/dL) among individuals with HbA_1c level >8.0% (64 mmol/mol), with greater reduction in HbA_1c (−1.67%) and LDL-C (−6.81 mg/dL) among those with HbA_1c level >9.5% at baseline (all P < 0.05). No significant impact on SBP and BMI was observed following the QI program.

Figure 2 shows the 5-year projected risk reduction of clinical outcomes following the QI intervention. Among patients with baseline HbA_1c >8% (64 mmol/mol), the model estimated the QI program to significantly reduce the 5-year risk of all-cause mortality (RR 0.83 [95% CI 0.82–0.85]), cardiovascular disease (CVD) death (RR 0.76 [95% CI 0.73–0.79]), nonfatal or fatal MI (RR 0.84 [95% CI 0.82–0.86]), nonfatal or fatal stroke (RR 0.73 [95% CI 0.69–0.78]), hospitalization for CHF (RR 0.92 [95% CI 0.89–0.95]), ESRD (RR 0.89 [95% CI 0.88–0.92]), major cardiovascular adverse event (RR 0.78 [95% CI 0.75–0.81]), blindness (RR 0.85 [95% CI 0.83–0.88]), and neuropathy (RR 0.78 [95% CI 0.75–0.81]).

Individuals with HbA_1c >9.5% (80 mmol/mol) at baseline experienced even larger predicted clinical benefits over 5 years following the QI intervention than those with baseline HbA_1c >8% (64 mmol/mol). In these patients, the model estimated the QI program to significantly reduce the 5-year risk of all-cause mortality (RR 0.61 [95% CI 0.59–0.64]), CVD death (RR 0.52 [95% CI 0.49–0.55]), nonfatal or fatal MI (RR 0.73 [95% CI 0.69–0.76]), nonfatal or fatal stroke (RR 0.56 [95% CI 0.51–0.62]), hospitalization for CHF (RR 0.85 [95% CI 0.81–0.90]), ESRD (RR 0.82 [95% CI 0.79–0.84]), major adverse cardiovascular event (MACE) (RR 0.60 [95% CI 0.56–0.65]), blindness (RR 0.74 [95% CI 0.71–0.78]), and neuropathy (RR 0.62 [95% CI 0.58–0.65]).

We have presented results of the sensitivity analysis in Supplementary Appendix 1. Under a conservative assumption on the persistency of the QI program’s clinical efficacy, the QI program is estimated to reduce the 5-year risk of all-cause mortality (RR 0.93 [95% CI 0.92–0.94]), CVD death (RR 0.90 [95% CI 0.88–0.91]), nonfatal or fatal MI (RR 0.94 [95% CI 0.93–0.95]), nonfatal or fatal stroke (RR 0.89 [95% CI 0.87–0.91]), hospitalization for CHF (RR 0.95 [95% CI 0.95–0.98]), ESRD (RR 0.96 [95% CI 0.94–0.96]), MACE (RR 0.91 [95% CI 0.90–0.92]), blindness (RR 0.94 [95% CI 0.94–0.96]), and neuropathy (RR 0.91 [95% CI 0.91–0.92]) among patients with baseline HbA_1c >8% (64 mmol/mol). The QI program can potentially achieve a larger predicted clinical benefits in individuals with HbA_1c >9.5% (80 mmol/mol) at baseline, with reductions of all-cause mortality (RR 0.82 [95% CI 0.80–0.83]), CVD death (RR 0.78 [95% CI 0.75–0.79]), nonfatal or fatal MI (RR 0.89 [95% CI 0.87–0.91]), nonfatal or fatal stroke (RR 0.81 [95% CI 0.78–0.84]), hospitalization for CHF (RR 0.94 [95% CI 0.92–0.96]), ESRD (RR 0.93 [95% CI 0.91–0.93]), MACE (RR 0.83 [95% CI 0.80–0.85]), blindness (RR 0.90 [95% CI 0.88–0.92]), and neuropathy (RR 0.84 [95% CI 0.83–0.86]) in 5 years.

Our results illustrate clinically significant projected reductions in diabetes-related outcomes and mortality in patients with poor glycemic control (i.e., HbA_1c >8% [64 mmol/mol]) following a QI program intervention such as that offered by Diabetes INSIDE. Our study identified a similar effect size on HbA_1c reduction as our previous study (8) and several studies conducted by others (17). Additionally, we examined the clinical efficacy of the QI program on individuals with very poor diabetes control (i.e., HbA_1c >9.5% [80 mmol/mol]) and found that the QI program is especially effective in these individuals, with ∼20% reduction (or −1.67% absolute change) in the HbA_1c level after a 1-year interval; these data can be further extrapolated into reductions in 5-year risks of diabetes-related complications by up to 48%. The substantial clinical benefit predicted for this poorly controlled group of patients with type 2 diabetes should translate into lower health care costs that should potentially offset the expense needed to deliver similar QI programs. A very limited HbA_1c reduction was observed among patients with a baseline HbA_1c of 8–9.5%, suggesting that the QI program might not benefit this subpopulation in terms of glucose control. However, a reduction of 5.08 mg/dL in LDL and 0.21 in BMI suggests that the QI program will likely reduce the risk of 5-year cardiovascular outcomes in this subpopulation through better control of lipids and body weights.

The diabetes simulation approach has been shown to be an important methodology to predict the longer-term program impact in the field of diabetes. In chronic conditions like diabetes, meaningful reductions in diabetes-related clinical outcomes following an intervention require years to manifest. Thus, researchers often use surrogate end points such as a change in HbA_1c over a relatively shorter time window to evaluate the effectiveness of an intervention (17–19). Unfortunately, surrogate end points or intermediate markers such as HbA_1c fail to provide evidence on risk reduction of diabetes complications. Thus, alternative methodologies need to be implemented to translate the clinical efficacy of surrogate end points achievement into risk reduction for diabetes-related complications. A diabetes simulation model that has been extensively validated against a large number of known trials is the key to generate evidence with a high level of validity. In this study, we chose to use the BRAVO diabetes model, because this model has been validated and calibrated against 18 recently conducted clinical trials (12), building evidence on the validity in using this model to draw clinical implications. We used a 5-year simulation experiment to extrapolate the clinical efficacy of the Diabetes INSIDE QI program on key biomarkers achieved at 1 year to predict the 5-year risk reduction in diabetes-related complications; this was based on the assumption that the impact of the QI program on biomarkers at year 1 will persist over the next 4 years, an assumption that has been validated by previous studies (14–16). Further, the nature of computerized EMR-based population health interventions is that they are easy to sustain long term. Individuals whose HbA_1c regresses would be identified again and appear periodically on their physicians’ dashboard as needing intervention, which could address the therapeutic inertia pervasive among health systems, including ours. Similar approaches have been successfully used in other studies (20). Nevertheless, the sustainability of the clinical benefit relied on how the QI program was implemented and how physicians adhere to the program’s protocol. Thus, to further understand the impact of the QI program, we have also conducted a sensitivity analysis and found that, under a diminishing yearly rate of 30% in achieving biomarker outcomes, the QI program can still result in a substantial risk reduction in cardiovascular outcomes (RR 0.81–0.94), microvascular outcomes (RR 0.84–0.93), and mortality (RR 0.82) in individuals with HbA_1c >9.5%.

To the best of our knowledge, this is the first study to evaluate the impact of a QI program for health management of the population with diabetes, which evaluated the 5-year direct risk reduction in diabetes-related complications based on surrogate outcomes such as HbA_1c goal achievement rates (19). We found that the ADA’s Diabetes INSIDE QI program can potentially lead to risk reductions in a number of cardiovascular, microvascular, and mortality outcomes. The effect size in risk reduction of complications varies from 48% (CVD death) to 8% (CHF), which is clinically meaningful and surprisingly large, considering the fact that the QI program targets disease-management workflows, instead of direct pharmacologic intervention. Our simulation experiment generates useful evidence on the long-term clinical impact of the Diabetes INSIDE QI program and thus could serve as a reference for designing a policy, with clear long-term objectives. A systematic literature review conducted by Nuckols et al. (21) has found that QI interventions that lower HbA_1c are likely to be of fair-to-good value relative to usual care, depending on the willingness of the corresponding society and policy environment to fund these efforts. While the incremental cost for implementing the ADA’s Diabetes INSIDE QI program was not examined in this study, we are optimistic that given the robust reduction in HbA_1c (−0.84% [and −1.74% among those with HbA_1c >9.5% (80 mmol/mol)]) compared with the study by Nuckols et al. (−0.26%), a QI program such as Diabetes INSIDE would be welcomed by clinicians and patients alike. The study has a few limitations. First, the pre-post study might suffer from time-related confounding issues. In other words, new, or changing, postintervention treatments may have a long-term health impact that biases our model estimates. Second, the sustainability of the QI program on biomarkers reduction is not clear. Although there is indirect evidence supporting the sustainability of such programs (14) over a 3-year period, it is also possible that the intervention effect could diminish over a longer time frame. The success of the QI program relies on the endurance of the program’s impact and requires persistent monitoring of population biomarkers to sustain improvement over time, including continuous QI. Third, our target population comprises a large proportion of minority groups, mostly Hispanics in Dallas and African Americans in New Orleans. This may also represent a strength of the intervention demonstrating success in lower socioeconomic and ethnic groups who are disproportionally affected by diabetes and its complications. Nevertheless, more studies are warranted to further explore the feasibility and impact of the QI program in other health systems and environments, covering populations with different clinical and demographic characteristics. It is possible that since QI is about care quality process improvement, it will naturally have lesser effects in healthier populations. Nevertheless, QI is about sustaining healthier populations. Fourth, while cost-effectiveness assessment is a vital piece to understand the long-term impact of launching the QI program, this was not an objective of the underlying projects, and data on health care utilization, drug costs, and hospitalizations were not routinely collected. Thus, attributions of cost reductions would have to make assumptions on costs that may not apply to the population studied.

We conclude that our outcomes model suggests that the observed risk factor reduction from the ADA Diabetes INSIDE QI program is projected to lead to clinically significant reductions in the 5-year risk of vascular events and other disease complications for high-risk patients and over time lessen the health burden for this patient population.

Funding. The original Diabetes INSIDE program was funded by a grant from the ADA to Tulane and Parkland. This analysis was not funded.

Duality of Interest. V.F. is supported in part by the Tullis-Tulane Alumni Chair in Diabetes. H.S., V.F., and L.S. codeveloped the BRAVO risk engine and have ownership interest in BRAVO4Health, a private company that aims to incorporate such risk assessment in clinical practice. V.F. has received research grants (to Tulane) from Bayer, Gilead Janssen, and Boehringer Ingelheim and received honoraria for consultation work from Novo Nordisk, Sanofi, Asahi Kasei Pharma, and AstraZeneca. L.M. has received honoraria for consultation work from Sanofi and Applied Therapeutics and has served on the advisory boards for Sanofi and Novo Nordisk. No other potential conflicts of interest relevant to this article were reported.

Author Contributions. All authors wrote the manuscript and researched data, contributed to discussion, and reviewed and edited the manuscript. V.F. is the guarantor of this work and, as such, had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.