OBJECTIVE

To construct and internally validate prediction models to estimate the risk of long-term end-organ complications and mortality in patients with type 2 diabetes and obesity that can be used to inform treatment decisions for patients and practitioners who are considering metabolic surgery.

RESEARCH DESIGN AND METHODS

A total of 2,287 patients with type 2 diabetes who underwent metabolic surgery between 1998 and 2017 in the Cleveland Clinic Health System were propensity-matched 1:5 to 11,435 nonsurgical patients with BMI ≥30 kg/m2 and type 2 diabetes who received usual care with follow-up through December 2018. Multivariable time-to-event regression and random forest machine learning models were built and internally validated using fivefold cross-validation to predict the 10-year risk for four outcomes of interest. The prediction models were programmed to construct user-friendly web-based and smartphone applications of Individualized Diabetes Complications (IDC) Risk Scores for clinical use.

RESULTS

The prediction tools demonstrated the following discrimination ability based on the area under the receiver operating characteristic curve (1 = perfect discrimination and 0.5 = chance) at 10 years in the surgical and nonsurgical groups, respectively: all-cause mortality (0.79 and 0.81), coronary artery events (0.66 and 0.67), heart failure (0.73 and 0.75), and nephropathy (0.73 and 0.76). When a patient’s data are entered into the IDC application, it estimates the individualized 10-year morbidity and mortality risks with and without undergoing metabolic surgery.

CONCLUSIONS

The IDC Risk Scores can provide personalized evidence-based risk information for patients with type 2 diabetes and obesity about future cardiovascular outcomes and mortality with and without metabolic surgery based on their current status of obesity, diabetes, and related cardiometabolic conditions.

Prevention of cardiovascular morbidity and mortality is a major therapeutic goal in type 2 diabetes. Several randomized clinical trials have shown the effectiveness of metabolic surgery (defined as procedures that influence metabolism by inducing weight loss and altering gastrointestinal physiology) for weight loss and control of type 2 diabetes in patients with obesity (13). Large observational studies have also reported a lower risk of microvascular and macrovascular complications of diabetes and mortality after metabolic surgery compared with usual care in patients with diabetes (410). The growing body of evidence on safety (11) and cardiometabolic benefits of metabolic surgery has led to inclusion of metabolic surgery in the treatment algorithm of type 2 diabetes in patients with obesity (12).

Despite reported benefits of metabolic surgery, according to most estimates, <1% of eligible patients undergo weight loss surgery (13). One potential explanation for underutilization is the lack of information on the treatment effects of metabolic surgery on health outcomes for individual patients. Optimized management in patients with type 2 diabetes should consider the likelihood of end-organ complications and mortality. However, tools for predicting the risk of end-organ complications of diabetes in individual patients, with and without metabolic surgery, are lacking. We recently reported the cardiovascular effects of metabolic surgery in a large cohort of patients with obesity and diabetes compared with patients who received usual diabetes care (4). The purpose of the current study was to identify the predictors of long-term major cardiovascular adverse events and death. We sought to construct a risk calculator that could be used to inform treatment decisions for patients and practitioners who are considering metabolic surgery.

The data source of this study was Cleveland Clinic’s electronic health record (EHR). The study protocol, definition of study variables, EHR codes, and construction of study cohorts have been recently published (4). The local Institutional Review Board approved the study as minimal risk research using data collected for routine clinical practice and permitted us to conduct the research without obtaining the informed consent from participants.

Briefly, a retrospective cohort of 287,438 patients with type 2 diabetes in the Cleveland Clinic Health System between 1998 and 2017 was used to identify 2,287 adult patients (age between 18 and 80 years old) with BMI ≥30 kg/m2 who underwent metabolic surgery. Exclusion criteria were history of solid organ transplant (liver, heart, or lung), history of severe heart failure (ejection fraction <20% any time before enrollment), and presence of active cancer. Application of enrollment criteria resulted in 39,267 nonsurgical patients with type 2 diabetes and obesity (BMI ≥30) before matching. Each surgical patient was propensity-matched by the nearest-neighbor method to five obese patients (BMI ≥30 kg/m2), resulting in 11,435 patients who received usual care. Matching criteria included the index date, age at index date, sex, BMI at index date, location of medical center, insulin use, and presence of diabetes-related end-organ complications (composite of coronary artery disease, heart failure, cerebrovascular disease, peripheral vascular disease, neuropathy, nephropathy, or requiring dialysis) with follow-up through December 2018. The matching procedure was designed to assemble a cohort of nonsurgical patients who would be eligible for (but did not receive) metabolic surgery at the index date.

Demographic characteristics, medical history, medication, and laboratory data were extracted from the EHR for each patient and used for analysis. Time-to-event outcomes included all-cause mortality, coronary artery events (unstable angina, myocardial infarction, or coronary intervention/surgery), heart failure, and nephropathy (Supplementary Table 1). Patients who already had experienced these adverse outcomes before the index date were removed from modeling for that specific outcome (e.g., heart failure in the heart failure model). Nephropathy was defined as the presence of at least two measures of estimated glomerular filtration rate (eGFR) <60 mL/min separated by at least 90 days without any intervening values ≥60 mL/min. The eGFR was approximated using the MDRD Study equation (14). Patients with a baseline eGFR <60 mL/min and patients with history of dialysis were excluded from the risk set of nephropathy outcome. Death information was retrieved from a combination of Cleveland Clinic’s EHR, Social Security, and state death indices.

Regression (Cox proportional hazards, exponential, and Fine-Gray) and random forest machine learning (cause-specific and competing risk) models were built with all variables in Table 1 as predictors, except for those with excessive missing data (defined as >25% in either treatment group, including HDL, LDL, and urinary albumin-to-creatinine ratio), impractical for future use (including index date, zip code income, and location), or rare (<1% in both groups: dialysis). Within each outcome data set, missing data were singly imputed using an iterative fully conditional approach (multivariate imputation by chained equations) (15). Full and stratified models were tested for each treatment group and internally validated by fivefold cross-validation.

Table 1

Characteristics of patients undergoing metabolic surgery and matched nonsurgical patients at the index date

Baseline variableMetabolic surgery (N = 2,287)Matched nonsurgical (N = 11,435)
Demographic data   
 Index date January 2013 (July 2010, April 2015) July 2013 (May 2011, April 2015) 
 Sex   
  Female 1,499 (65.5) 7,339 (64.2) 
  Male 788 (34.5) 4,096 (35.8) 
 Age (years) 52.5 (43.7, 60.5) 54.8 (46.2, 62.5) 
 BMI (kg/m245.1 (40, 51.8) 42.6 (39.4, 47.2) 
 BMI category (kg/m2  
  30–34.9 109 (4.8) 495 (4.3) 
  35–39.9 465 (20.3) 2,595 (22.7) 
  ≥40 1,713 (74.9) 8,345 (73.0) 
 Race   
  White 1,734 (75.8) 7,994 (69.9) 
  Black 441 (19.3) 2,804 (24.5) 
  Other 54 (2.4) 234 (2.1) 
  Missing 58 (2.5) 403 (3.5) 
 Annual zip code income ($) 49,855 (39,964, 62,273) 48,732 (36,951, 61,512) 
  Missing 70 (3.1) 125 (1.1) 
 Smoking status   
  Never 1,231 (53.8) 5,615 (49.1) 
  Former 828 (36.2) 4,012 (35.1) 
  Current 170 (7.4) 1,607 (14) 
  Missing 58 (2.5) 201 (1.8) 
 Location   
  Ohio 1,816 (79.1) 9,834 (86.0) 
  Florida 471 (20.6) 1,601 (14.0) 
Medical history   
 Hypertension 1,953 (85.4) 8,565 (74.9) 
 Dyslipidemia 1,686 (73.7) 7,457 (65.2) 
 Peripheral neuropathy 242 (10.6) 1,203 (10.5) 
 Heart failure 238 (10.4) 1,342 (11.7) 
 Coronary artery disease 237 (10.4) 1,104 (9.7) 
 COPD 206 (9.0) 1,188 (10.4) 
 Nephropathy 191 (8.4) 1,219 (10.7) 
 Peripheral arterial disease 123 (5.4) 755 (6.6) 
 Cerebrovascular disease 42 (1.8) 358 (3.1) 
 Dialysis 14 (0.6) 78 (0.7) 
Clinical and laboratory data   
 HbA1c (%) 7.1 (6.3, 8.2) 7.1 (6.4, 8.4) 
 HbA1c (mmol/mol) 54 (45, 66) 54 (46, 68) 
  Missing 159 (7.0) 1,288 (11.3) 
 Systolic blood pressure (mmHg) 138 (127, 148.43) 130.2 (121, 142) 
  Missing — 54 (0.5) 
 Diastolic blood pressure (mmHg) 71.5 (65, 79) 78 (70, 84) 
  Missing — 54 (0.5) 
 eGFR (mL/min/1.73 m2)a 90.3 (72.3, 108.3) 91.9 (72.5, 111.9) 
  Missing 1 (0) 432 (3.8) 
 HDL (mg/dL) 44 (37, 52) 43 (36, 51) 
  Missing 792 (34.6) 3,512 (30.7) 
 LDL (mg/dL) 92 (72, 115) 93 (72, 118) 
  Missing 208 (9.1) 2,947 (25.8) 
 Triglycerides (mg/dL) 146 (101, 209) 146 (103, 208) 
  Missing 160 (7) 1,816 (15.9) 
 UACR (mg/g) 14 (5.7, 40.1) 14 (5, 43) 
  Missing 1,003 (43.9) 4,224 (36.9) 
Medication history   
 Noninsulin diabetes medications 1,869 (81.7) 9,253 (80.9) 
 Insulin 776 (33.9) 3,806 (33.3) 
 Lipid-lowering medications 1,195 (52.3) 5,998 (52.5) 
 Renin-angiotensin system inhibitorsb 1,396 (61.0) 7,102 (62.1) 
 Other antihypertensive medications 1,649 (72.1) 8,066 (70.5) 
 Aspirin 731 (32.0) 4,627 (40.5) 
 Warfarin 190 (8.3) 943 (8.2) 
Baseline variableMetabolic surgery (N = 2,287)Matched nonsurgical (N = 11,435)
Demographic data   
 Index date January 2013 (July 2010, April 2015) July 2013 (May 2011, April 2015) 
 Sex   
  Female 1,499 (65.5) 7,339 (64.2) 
  Male 788 (34.5) 4,096 (35.8) 
 Age (years) 52.5 (43.7, 60.5) 54.8 (46.2, 62.5) 
 BMI (kg/m245.1 (40, 51.8) 42.6 (39.4, 47.2) 
 BMI category (kg/m2  
  30–34.9 109 (4.8) 495 (4.3) 
  35–39.9 465 (20.3) 2,595 (22.7) 
  ≥40 1,713 (74.9) 8,345 (73.0) 
 Race   
  White 1,734 (75.8) 7,994 (69.9) 
  Black 441 (19.3) 2,804 (24.5) 
  Other 54 (2.4) 234 (2.1) 
  Missing 58 (2.5) 403 (3.5) 
 Annual zip code income ($) 49,855 (39,964, 62,273) 48,732 (36,951, 61,512) 
  Missing 70 (3.1) 125 (1.1) 
 Smoking status   
  Never 1,231 (53.8) 5,615 (49.1) 
  Former 828 (36.2) 4,012 (35.1) 
  Current 170 (7.4) 1,607 (14) 
  Missing 58 (2.5) 201 (1.8) 
 Location   
  Ohio 1,816 (79.1) 9,834 (86.0) 
  Florida 471 (20.6) 1,601 (14.0) 
Medical history   
 Hypertension 1,953 (85.4) 8,565 (74.9) 
 Dyslipidemia 1,686 (73.7) 7,457 (65.2) 
 Peripheral neuropathy 242 (10.6) 1,203 (10.5) 
 Heart failure 238 (10.4) 1,342 (11.7) 
 Coronary artery disease 237 (10.4) 1,104 (9.7) 
 COPD 206 (9.0) 1,188 (10.4) 
 Nephropathy 191 (8.4) 1,219 (10.7) 
 Peripheral arterial disease 123 (5.4) 755 (6.6) 
 Cerebrovascular disease 42 (1.8) 358 (3.1) 
 Dialysis 14 (0.6) 78 (0.7) 
Clinical and laboratory data   
 HbA1c (%) 7.1 (6.3, 8.2) 7.1 (6.4, 8.4) 
 HbA1c (mmol/mol) 54 (45, 66) 54 (46, 68) 
  Missing 159 (7.0) 1,288 (11.3) 
 Systolic blood pressure (mmHg) 138 (127, 148.43) 130.2 (121, 142) 
  Missing — 54 (0.5) 
 Diastolic blood pressure (mmHg) 71.5 (65, 79) 78 (70, 84) 
  Missing — 54 (0.5) 
 eGFR (mL/min/1.73 m2)a 90.3 (72.3, 108.3) 91.9 (72.5, 111.9) 
  Missing 1 (0) 432 (3.8) 
 HDL (mg/dL) 44 (37, 52) 43 (36, 51) 
  Missing 792 (34.6) 3,512 (30.7) 
 LDL (mg/dL) 92 (72, 115) 93 (72, 118) 
  Missing 208 (9.1) 2,947 (25.8) 
 Triglycerides (mg/dL) 146 (101, 209) 146 (103, 208) 
  Missing 160 (7) 1,816 (15.9) 
 UACR (mg/g) 14 (5.7, 40.1) 14 (5, 43) 
  Missing 1,003 (43.9) 4,224 (36.9) 
Medication history   
 Noninsulin diabetes medications 1,869 (81.7) 9,253 (80.9) 
 Insulin 776 (33.9) 3,806 (33.3) 
 Lipid-lowering medications 1,195 (52.3) 5,998 (52.5) 
 Renin-angiotensin system inhibitorsb 1,396 (61.0) 7,102 (62.1) 
 Other antihypertensive medications 1,649 (72.1) 8,066 (70.5) 
 Aspirin 731 (32.0) 4,627 (40.5) 
 Warfarin 190 (8.3) 943 (8.2) 

Data are n (%) or median (interquartile range).

COPD, chronic obstructive pulmonary disease; UACR, urinary albumin-to-creatinine ratio.

a

eGFR was approximated using the MDRD Study equation (14).

b

Including ACE inhibitors and angiotensin receptor blockers.

Adapted and modified from Aminian et al. (4).

Prediction model performance was primarily assessed with the index of prediction accuracy (IPA) that combines discrimination and calibration in one value, ranging from 1 to −1, with higher IPAs indicating better performance; IPA = 1 is a perfect model, IPA = 0 is a useless model, and harmful models have an IPA <0 (16). Discrimination was also assessed with the time-dependent area under the receiver operating characteristic curve (AUC), and calibration was evaluated graphically by plotting the quintiles of the predicted risk against the actual outcome proportion. The closer the points lie along the 45° line, the better the calibration. For each outcome and treatment combination, the modeling framework (either regression or random forest) producing the largest cross-validated IPA at 10 years was chosen. The R package “riskRegression” was used (17).

The relative importance computed for each predictor in the final model depended on the modeling framework. For regression models, variables were ranked by the increase in Akaike information criterion after removal from the full model. For random forest machine learning models, variables were ranked by the decrease in C-index for out-of-bag data after random permutation.

Further assessment of model performance was performed by using data of 27,832 eligible nonsurgical patients who were not matched in the 1:5 matching process. Performance of Individualized Diabetes Complications (IDC) Risk Scores equations for nonsurgical groups was compared with that of the Risk Equations for Complications of Type 2 Diabetes (RECODe) (18,19) by comparing IPA, time-dependent AUC, and model calibration for the outcomes of all-cause mortality, heart failure, and nephropathy. RECODe equations (18,19) were originally derived from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) randomized trial (20) and validated with the Diabetes Prevention Program Outcomes Study (DPPOS) (21) and Look AHEAD (Action for Health in Diabetes) randomized trials (22). It has been shown that RECODe equations have better discrimination and calibration as compared with the UK Prospective Diabetes Study Outcomes Model 2 (UKPDS OM2) (23). Among 27,832 nonsurgical patients who were not included in the training data set, 12,816 patients had all required data for the IDC Risk Scores and RECODe formulas. Although the definitions were not exactly the same, an exploratory comparison between two models was performed for nephropathy outcome. Per the ACCORD trial, nephropathy-3 outcome was defined as development of renal failure: end-stage renal disease, or initiation of dialysis, or rise of serum creatinine >3.3 mg/dL in absence of an acute reversible cause. Because the IDC Risk Scores estimate the incident coronary artery events (including unstable angina, myocardial infarction, or coronary intervention/surgery), and RECODe equations predict myocardial infarction only, comparison of model performance was not performed for that outcome.

The prediction models were programmed in user-friendly web-based and smartphone applications of IDC Risk Scores for clinical use.

Baseline characteristics of 13,722 patients, including 2,287 patients undergoing metabolic surgery and 11,435 matched nonsurgical patients, are presented in Table 1. At enrollment, 73% had a BMI ≥40, 22% had a BMI between 35 and 39.9, and 4% had a BMI between 30 and 34.9 kg/m2. Metabolic surgical procedures included Roux-en-Y gastric bypass (n = 1,443; 63%), sleeve gastrectomy (n = 730; 32%), adjustable gastric banding (n = 109; 5%), and duodenal switch (n = 5).

In the metabolic surgery group, 65% were female with a median age of 52.5 years, BMI of 45.1 kg/m2, HbA1c level of 7.1% (54 mmol/mol), and eGFR of 90.3 mL/min. The percentages of patients taking insulin and noninsulin diabetes medications were 34% and 82%, respectively. In the nonsurgical group, 64% were female, with a median age of 54.8 years, BMI of 42.6 kg/m2, HbA1c level of 7.1% (54 mmol/mol), and eGFR of 91.9 mL/min. The percentages of patients taking insulin and noninsulin diabetes medications were 33% and 81%, respectively.

In a median follow-up time of 3.9 years (interquartile range 1.9–6.1), 112 patients in the metabolic surgery group and 1,111 patients in the nonsurgical group died. At the end of study, in the surgical group and nonsurgical group, 84 and 676 patients had coronary artery events, 76 and 1,055 patients had heart failure, and 69 and 810 patients had nephropathy, respectively. Comparisons of adjusted incidence of these outcomes across groups have previously been published (4).

Random forest models had a slightly higher 10-year IPA in two of eight models, including coronary artery events in the surgical patients and nephropathy in the nonsurgical patients. Regression models were chosen for the other six models. The largest cross-validated IPAs at 10 years were 0.13 and 0.24 for all-cause mortality, 0.03 and 0.04 for coronary artery events, 0.05 and 0.14 for heart failure, and 0.07 and 0.14 for nephropathy in the surgical and nonsurgical groups, respectively (Table 2).

Table 2

Performance of IDC Risk Scores on training data set and comparison with the RECODe models for nonsurgical patients

OutcomeGroupModel comparison*
Performance of IDC Risk ScoresIDCRECODe
ModelIPAAUCIPAAUCIPAAUC
All-cause mortality Metabolic surgery Regression 0.13 0.79 — — — — 
All-cause mortality Usual care Regression 0.24 0.81 0.20 0.78 0.12 0.76 
Coronary artery events Metabolic surgery Random forest 0.03 0.66 — — — — 
Coronary artery events Usual care Regression 0.04 0.67 — — — — 
Heart failure Metabolic surgery Regression 0.05 0.73 — — — — 
Heart failure Usual care Regression 0.14 0.75 0.15 0.75 0.004 0.73 
Nephropathy Metabolic surgery Regression 0.07 0.73 — — — — 
Nephropathy Usual care Random forest 0.14 0.76 0.18 0.77 −0.19 0.60 
OutcomeGroupModel comparison*
Performance of IDC Risk ScoresIDCRECODe
ModelIPAAUCIPAAUCIPAAUC
All-cause mortality Metabolic surgery Regression 0.13 0.79 — — — — 
All-cause mortality Usual care Regression 0.24 0.81 0.20 0.78 0.12 0.76 
Coronary artery events Metabolic surgery Random forest 0.03 0.66 — — — — 
Coronary artery events Usual care Regression 0.04 0.67 — — — — 
Heart failure Metabolic surgery Regression 0.05 0.73 — — — — 
Heart failure Usual care Regression 0.14 0.75 0.15 0.75 0.004 0.73 
Nephropathy Metabolic surgery Regression 0.07 0.73 — — — — 
Nephropathy Usual care Random forest 0.14 0.76 0.18 0.77 −0.19 0.60 
*

Assessment of nonsurgical models on 12,816 patients at the Cleveland Clinic who were not included in the training data set was performed by direct comparison of IDC Risk Scores and the RECODe formulas (18,19). The IDC Risk Scores outperformed RECODe for all three examined outcomes (mortality, heart failure, and nephropathy) in terms of IPA, AUC, and calibration.

The prediction tools demonstrated the following discrimination ability based on the AUC (1 = perfect discrimination and 0.5 = chance) at 10 years in the surgical and nonsurgical groups, respectively: all-cause mortality (0.79 and 0.81), coronary artery events (0.66 and 0.67), heart failure (0.73 and 0.75), and nephropathy (0.73 and 0.76).

The calibration of cross-validated 10-year risks for each outcome stratified by treatment group is shown in Fig. 1. The relative contribution of each variable in explaining the outcomes in final models is shown in Fig. 2 and Supplementary Table 2. The five most important variables in the prediction models of all-cause mortality were age, BMI, history of heart failure, insulin use, and smoking status.

Figure 1

Calibration of cross-validated 10-year risks for each outcome, stratified by treatment group. Cross-validated risks were binned into 10 subgroups, and each subgroup’s average risk was plotted against the observed cumulative incidence of the subgroup (Kaplan-Meier for all-cause mortality). Local regression smoothers were then drawn through the points. The closer the points lie along the 45° line, the better the calibration.

Figure 1

Calibration of cross-validated 10-year risks for each outcome, stratified by treatment group. Cross-validated risks were binned into 10 subgroups, and each subgroup’s average risk was plotted against the observed cumulative incidence of the subgroup (Kaplan-Meier for all-cause mortality). Local regression smoothers were then drawn through the points. The closer the points lie along the 45° line, the better the calibration.

Close modal
Figure 2

The relative rankings of importance of each baseline variable in final prediction models for each outcome (all-cause mortality [A], coronary artery disease [B], heart failure [C], nephropathy [D]) stratified by treatment group. For regression models (six of eight models depicted in gray), the increase in Akaike information criterion (AIC) upon removal from the full model was computed for each variable and ranked. For random forest models (two of eight models depicted in orange, including coronary artery events in the surgical patients and nephropathy in the nonsurgical patients), a permutation technique was used by comparing the out-of-bag prediction accuracy (based on the C-index) of the original forest with that after randomly permuting each variable. COPD, chronic obstructive pulmonary disease.

Figure 2

The relative rankings of importance of each baseline variable in final prediction models for each outcome (all-cause mortality [A], coronary artery disease [B], heart failure [C], nephropathy [D]) stratified by treatment group. For regression models (six of eight models depicted in gray), the increase in Akaike information criterion (AIC) upon removal from the full model was computed for each variable and ranked. For random forest models (two of eight models depicted in orange, including coronary artery events in the surgical patients and nephropathy in the nonsurgical patients), a permutation technique was used by comparing the out-of-bag prediction accuracy (based on the C-index) of the original forest with that after randomly permuting each variable. COPD, chronic obstructive pulmonary disease.

Close modal

Further assessment of nonsurgical models on 12,816 patients of the Cleveland Clinic who were not included in the training data set was performed by direct comparison of IDC Risk Scores and RECODe formulas. The IDC Risk Scores outperformed RECODe on all examined outcomes in terms of IPA, AUC, and calibration (Table 2 and Supplementary Fig. 1). The IPAs at 10 years for the IDC Risk Scores and RECODe models were 0.20 and 0.12 for all-cause mortality, 0.15 and 0.004 for heart failure, and 0.18 and −0.19 for nephropathy, respectively.

The final models were integrated into the IDC risk calculator, which allows entry of patient information to obtain individualized predictions of each outcome with and without undergoing metabolic surgery. For example, for a 50-year-old African American female patient with a BMI of 35 kg/m2, glycated hemoglobin of 7.8%, blood pressure of 130/70 mmHg, serum creatinine of 1 mg/dL, and triglycerides of 150 mg/dL, without history of macrovascular and microvascular complications from diabetes, who is taking statin for dyslipidemia and insulin, the models predict a 10.9% risk of all-cause mortality, 7.7% risk of coronary artery events, 12.2% risk of heart failure, and 24.8% risk of diabetic nephropathy with usual care in 10 years. After metabolic surgery, the estimated 10-year risk of these adverse events is 5.6%, 5.6%, 4.2%, and 8%, respectively.

Using long-term data on cardiovascular outcomes of metabolic surgery and nonsurgical patients and rigorous statistical techniques, accurate prediction models were constructed and integrated into a risk calculator to estimate the likelihood of long-term end-organ complications and death in patients with type 2 diabetes and obesity. The area under the curve of prediction models ranged from 0.66 to 0.81, with the best discrimination ability for all-cause mortality. The calibration curves indicated that the prediction models do not substantially overestimate or underestimate risk. Direct comparison of IDC Risk Scores with RECODe models on a separate cohort of nonsurgical patients suggested better performance of new models.

Although risk factors for cardiovascular morbidity and mortality in patients with type 2 diabetes have been widely described, there are few published prediction models. The most widely used diabetes-specific prediction model is the UKPDS risk engine. It estimates 10-year risk of coronary artery events and stroke from a noncontemporary cohort of ∼5,000 patients with type 2 diabetes without heart disease who were enrolled between 1977 and 1991. Ten predictive factors included age, sex, ethnicity, smoking status, duration of diabetes, HbA1c, systolic blood pressure, total cholesterol, HDL cholesterol, and atrial fibrillation (24,25). However, the validity of this model for patients with diabetes treated with contemporary medications remains uncertain.

The available prediction models to estimate the future risk of macrovascular complications, microvascular adverse events, and mortality in patients with type 2 diabetes have some limitations (18,2434). They usually only address single outcome (mainly risk of all-cause mortality) and do not simultaneously provide estimates for multiple clinically important end points for individual patients with type 2 diabetes and obesity. Furthermore, most available models use a limited number of baseline variables to estimate the risk for a shorter time horizon (i.e., 5-year risk prediction) in patients without major baseline comorbidities. In the current study, 26 diverse baseline variables, including demographic data, history of medical comorbidities, laboratory data, and medications, were considered in our models to predict the 10-year risk. More importantly, the common statistical technique used for construction of available prediction models in the literature has been the Cox proportional hazards modeling. To develop the IDC risk scores, in addition to classical regression models (including the Cox proportional hazards), we used a machine learning approach and built random forest prediction models. The machine learning–derived risk-prediction models yielded favorable discrimination and larger cross-validated IPA at 10 years compared with traditional regression models for two of the eight outcomes of interest. In addition, to the best of our knowledge, there is no other prediction model to estimate future risk of end-organ complications of diabetes and obesity after metabolic surgery.

These prediction models may be clinically useful for health care professionals, including primary care physicians, endocrinologists, cardiologists, and bariatric surgeons, and patients with type 2 diabetes and obesity to provide evidence-based predictions of the 10-year risk of adverse cardiovascular outcomes based on the current health status. The calculator uses clinical and laboratory information that is readily available in clinical practice.

A web version of the IDC Risk Scores is accessible at https://riskcalc.org (under the Obesity menu) and as a smartphone application (BariatricCalc). When the required values are entered into the calculator, the IDC Risk Scores are computed to assist in shared decision making for patients who are considering metabolic surgery for management of their obesity and diabetes. Particularly, the models would be clinically useful to identify individuals who have a high risk of death and major end-organ complications of diabetes with continuation of usual care. Within this group, if the predicted risk estimates are substantially lower after metabolic surgery, a decision to undergo surgical intervention might represent an opportunity for improved patient management. Conversely, when the risk estimates of adverse cardiovascular events are relatively comparable or higher after surgical intervention, continuation of usual care might be recommended. The IDC Risk Scores potentially address physician and patient concerns in preventing a wide range of life-threatening adverse events of diabetes and obesity.

This study has several limitations. First, the aim of this study was to build prediction models to optimally discriminate the outcomes of interest. The aim was not to build causal models. The rank of each variable in Fig. 2 does not necessarily show the importance of that variable in the chain of causation. For example, use of aspirin as one of the most important variables in the models of coronary artery events does not mean aspirin causes coronary artery events. Second, data from the EHR were retrospectively obtained, which would be associated with coding errors and misclassification. Furthermore, although a comprehensive propensity matching was performed, residual measured or unmeasured confounders could have influenced the outcomes. To develop prediction models, a large sample size is needed, which was made possible using a retrospective design. Similar to previous studies, the mortality risk models had the best discrimination ability among our models. That is probably due to the fact that mortality is a well-defined outcome with minimal risk of misclassification. Third, some possible important variables, including duration of diabetes, family history of coronary artery disease, LDL, HDL, and urinary albumin-to-creatinine ratio, were missing from too many patients and could not be included in the models. Fourth, because 95% of surgical patients underwent either gastric bypass or sleeve gastrectomy, the predictive surgical models may not be generalizable to other metabolic surgical procedures. Nonetheless, these two procedures are the two most common metabolic procedures in the current practice. Fifth, relatively few nonsurgical patients were exposed to newer classes of diabetes medications that are associated with significant cardiovascular benefits. In a similar fashion, 5% of the surgical group had gastric banding, which is the least effective surgical procedure for type 2 diabetes. Ideally, the prediction models should be periodically updated in order to estimate the risk for contemporary medical and surgical treatments, and not for old suboptimal therapies. Sixth, development of prediction models to estimate short- and long-term major complications of metabolic surgery would be the subject of future projects. Seventh, although internal cross-validation and comparison with the RECODe models were done, lack of an external validation cohort and evidence on clinical usefulness of prediction models are other limitations. However, it is challenging to find appropriate cohorts with adequate follow-up time to assess cardiovascular outcomes after metabolic surgery for external validation. Despite these limitations, this was the first effort to identify predictors and construct a risk calculator using both traditional regression methods and a machine learning approach with sufficient accuracy in calibration and discrimination to predict long-term cardiovascular outcomes after metabolic surgery.

In conclusion, this study showed that major adverse cardiovascular events (all-cause mortality, coronary artery events, heart failure, and nephropathy) are predictable outcomes in patients with type 2 diabetes and obesity who undergo metabolic surgery or receive usual diabetes care. The IDC Risk Scores can provide personalized evidence-based risk information for patients with type 2 diabetes and obesity about future cardiovascular outcomes and mortality with and without metabolic surgery, based on their current status of obesity, diabetes, and related cardiometabolic conditions. The prediction models would benefit from external validation.

See accompanying article, p. 701.

Acknowledgments. The authors thank Alex Milinovich and Jian Jin from the Department of Quantitative Health Sciences, Cleveland Clinic (Cleveland, OH) for creation of the database.

Funding and Duality of Interest. This study was partially funded by an unrestricted grant from Medtronic. A.A. reports receiving grants from Medtronic. D.E.A. reports receiving grants from the National Institutes of Health National Institute of Diabetes and Digestive and Kidney Diseases (R01-DK-105960) and the Patient-Centered Outcomes Research Institute and receiving nonfinancial support from the International Federation for the Surgery of Obesity and Metabolic Disorders, Latin America Chapter. S.A.B. reports receiving grants from Medtronic and GI Windows. P.R.S. reports receiving grants from Medtronic, Ethicon, and Pacira and receiving personal fees from Medtronic, GI Dynamics, W.L. Gore and Associates, Becton Dickinson Surgical, and Global Academy for Medical Education. S.E.N. reports receiving a grant from Medtronic for the current study and receiving research support from Amgen, AbbVie, AstraZeneca, Cerenis, Eli Lilly, Esperion Therapeutics, Novo Nordisk, The Medicines Company, Orexigen, Pfizer, and Takeda and consulting for a number of pharmaceutical companies without financial compensation (all honoraria, consulting fees, or other payments from any for-profit entity are paid directly to charity, so neither income nor any tax deduction is received). M.W.K. reports receiving grants from Medtronic and Novo Nordisk. No other potential conflicts of interest relevant to this article were reported.

Medtronic had no role in the design and conduct of the study; collection, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.

Author Contributions. All authors were responsible for study concept and design. A.A., A.Z., and M.W.K. were responsible for acquisition and analysis of data. A.Z. was responsible for statistical analysis. All authors were responsible for interpretation of data. A.A. was responsible for drafting of the manuscript. All authors conducted critical revision of the manuscript. A.A. obtained funding. A.A., S.E.N., and M.W.K. were responsible for supervision. A.A. and A.Z. are the guarantors of this work and, as such, had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Prior Presentation. This study was presented as one of the Top Papers at the ObesityWeek 2019 Congress, Las Vegas, NV, 3–7 November 2019.

1.
Ikramuddin
S
,
Korner
J
,
Lee
WJ
, et al
.
Lifestyle intervention and medical management with vs without Roux-en-Y gastric bypass and control of hemoglobin A1c, LDL cholesterol, and systolic blood pressure at 5 years in the Diabetes Surgery Study
.
JAMA
2018
;
319
:
266
278
2.
Mingrone
G
,
Panunzi
S
,
De Gaetano
A
, et al
.
Bariatric-metabolic surgery versus conventional medical treatment in obese patients with type 2 diabetes: 5 year follow-up of an open-label, single-centre, randomised controlled trial
.
Lancet
2015
;
386
:
964
973
3.
Schauer
PR
,
Bhatt
DL
,
Kirwan
JP
, et al.;
STAMPEDE Investigators
.
Bariatric surgery versus intensive medical therapy for diabetes - 5-year outcomes
.
N Engl J Med
2017
;
376
:
641
651
4.
Aminian
A
,
Zajichek
A
,
Arterburn
DE
, et al
.
Association of metabolic surgery with major adverse cardiovascular outcomes in patients with type 2 diabetes and obesity
.
JAMA
2019
;
322
:
1271
1282
5.
Sjöström
L
,
Peltonen
M
,
Jacobson
P
, et al
.
Association of bariatric surgery with long-term remission of type 2 diabetes and with microvascular and macrovascular complications
.
JAMA
2014
;
311
:
2297
2304
6.
Fisher
DP
,
Johnson
E
,
Haneuse
S
, et al
.
Association between bariatric surgery and macrovascular disease outcomes in patients with type 2 diabetes and severe obesity
.
JAMA
2018
;
320
:
1570
1582
7.
Adams
TD
,
Gress
RE
,
Smith
SC
, et al
.
Long-term mortality after gastric bypass surgery
.
N Engl J Med
2007
;
357
:
753
761
8.
Carlsson
LMS
,
Sjöholm
K
,
Karlsson
C
, et al
.
Long-term incidence of microvascular disease after bariatric surgery or usual care in patients with obesity, stratified by baseline glycaemic status: a post-hoc analysis of participants from the Swedish Obese Subjects study
.
Lancet Diabetes Endocrinol
2017
;
5
:
271
279
9.
O’Brien
R
,
Johnson
E
,
Haneuse
S
, et al
.
Microvascular outcomes in patients with diabetes after bariatric surgery versus usual care: a matched cohort study
.
Ann Intern Med
2018
;
169
:
300
310
10.
Aminian
A
,
Aleassa
EM
,
Bhatt
DL
, et al
.
Bariatric surgery is associated with a lower rate of death after myocardial infarction and stroke: a nationwide study
.
Diabetes Obes Metab
2019
;
21
:
2058
2067
11.
Aminian
A
,
Brethauer
SA
,
Kirwan
JP
,
Kashyap
SR
,
Burguera
B
,
Schauer
PR
.
How safe is metabolic/diabetes surgery?
Diabetes Obes Metab
2015
;
17
:
198
201
12.
Rubino
F
,
Nathan
DM
,
Eckel
RH
, et al.;
Delegates of the 2nd Diabetes Surgery Summit
.
Metabolic surgery in the treatment algorithm for type 2 diabetes: a joint statement by international diabetes organizations
.
Diabetes Care
2016
;
39
:
861
877
13.
Campos
GM
,
Khoraki
J
,
Browning
MG
,
Pessoa
BM
,
Mazzini
GS
,
Wolfe
L
.
Changes in utilization of bariatric surgery in the United States from 1993 to 2016
.
Ann Surg
2020
;
271
:
201
209
14.
Levey
AS
,
Bosch
JP
,
Lewis
JB
,
Greene
T
,
Rogers
N
,
Roth
D
;
Modification of Diet in Renal Disease Study Group
.
A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation
.
Ann Intern Med
1999
;
130
:
461
470
15.
Azur
MJ
,
Stuart
EA
,
Frangakis
C
,
Leaf
PJ
.
Multiple imputation by chained equations: what is it and how does it work?
Int J Methods Psychiatr Res
2011
;
20
:
40
49
16.
Kattan
MW
,
Gerds
TA
.
The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models
.
Diagn Progn Res
2018
;
2
:
7
17.
Package ‘riskRegression’. Risk Regression Models and Prediction Scores for Survival Analysis with Competing Risks
.
Version 2019.01.29 [Internet]
.
Available from project.org/web/packages/riskRegression/riskRegression.pdf. Accessed 1 October 2019
18.
Basu
S
,
Sussman
JB
,
Berkowitz
SA
,
Hayward
RA
,
Yudkin
JS
.
Development and validation of Risk Equations for Complications Of type 2 Diabetes (RECODe) using individual participant data from randomised trials
.
Lancet Diabetes Endocrinol
2017
;
5
:
788
798
19.
Basu
S
,
Sussman
JB
,
Berkowitz
SA
, et al
.
Validation of risk equations for complications of type 2 diabetes (RECODe) using individual participant data from diverse longitudinal cohorts in the U.S
.
Diabetes Care
2018
;
41
:
586
595
20.
Gerstein
HC
,
Miller
ME
,
Byington
RP
, et al.;
Action to Control Cardiovascular Risk in Diabetes Study Group
.
Effects of intensive glucose lowering in type 2 diabetes
.
N Engl J Med
2008
;
358
:
2545
2559
21.
Diabetes Prevention Program Research Group
.
Long-term effects of lifestyle intervention or metformin on diabetes development and microvascular complications over 15-year follow-up: the Diabetes Prevention Program Outcomes Study
.
Lancet Diabetes Endocrinol
2015
;
3
:
866
875
22.
Wing
RR
,
Bolin
P
,
Brancati
FL
, et al.;
Look AHEAD Research Group
.
Cardiovascular effects of intensive lifestyle intervention in type 2 diabetes
.
N Engl J Med
2013
;
369
:
145
154
23.
Hayes
AJ
,
Leal
J
,
Gray
AM
,
Holman
RR
,
Clarke
PM
.
UKPDS outcomes model 2: a new version of a model to simulate lifetime health outcomes of patients with type 2 diabetes mellitus using data from the 30 year United Kingdom Prospective Diabetes Study: UKPDS 82
.
Diabetologia
2013
;
56
:
1925
1933
24.
Kothari
V
,
Stevens
RJ
,
Adler
AI
, et al
.
UKPDS 60: risk of stroke in type 2 diabetes estimated by the UK Prospective Diabetes Study risk engine
.
Stroke
2002
;
33
:
1776
1781
25.
Stevens
RJ
,
Kothari
V
,
Adler
AI
,
Stratton
IM
;
United Kingdom Prospective Diabetes Study (UKPDS) Group
.
The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56)
.
Clin Sci (Lond)
2001
;
101
:
671
679
26.
Segar
MW
,
Vaduganathan
M
,
Patel
KV
, et al
.
Machine learning to predict the risk of incident heart failure hospitalization among patients with diabetes: the WATCH-DM risk score
.
Diabetes Care
2019
;
42
:
2298
2306
27.
Bergmark
BA
,
Bhatt
DL
,
Braunwald
E
, et al
.
Risk assessment in patients with diabetes with the TIMI risk score for atherothrombotic disease
.
Diabetes Care
2018
;
41
:
577
585
28.
Copetti
M
,
Shah
H
,
Fontana
A
, et al
.
Estimation of mortality risk in type 2 diabetic patients (ENFORCE): an inexpensive and parsimonious prediction model
.
J Clin Endocrinol Metab
2019
;
104
:
4900
4908
29.
Wells
BJ
,
Jain
A
,
Arrigain
S
,
Yu
C
,
Rosenkrans
WA
 Jr
,
Kattan
MW
.
Predicting 6-year mortality risk in patients with type 2 diabetes
.
Diabetes Care
2008
;
31
:
2301
2306
30.
Wells
BJ
,
Roth
R
,
Nowacki
AS
, et al
.
Prediction of morbidity and mortality in patients with type 2 diabetes
.
PeerJ
2013
;
1
:
e87
31.
Robinson
TE
,
Elley
CR
,
Kenealy
T
,
Drury
PL
.
Development and validation of a predictive risk model for all-cause mortality in type 2 diabetes
.
Diabetes Res Clin Pract
2015
;
108
:
482
488
32.
De Cosmo
S
,
Copetti
M
,
Lamacchia
O
, et al
.
Development and validation of a predicting model of all-cause mortality in patients with type 2 diabetes
.
Diabetes Care
2013
;
36
:
2830
2835
33.
McEwen
LN
,
Karter
AJ
,
Waitzfelder
BE
, et al
.
Predictors of mortality over 8 years in type 2 diabetic patients: Translating Research Into Action for Diabetes (TRIAD)
.
Diabetes Care
2012
;
35
:
1301
1309
34.
Tanaka
S
,
Tanaka
S
,
Iimuro
S
, et al.;
Japan Diabetes Complications Study Group
;
Japanese Elderly Diabetes Intervention Trial Group
.
Predicting macro- and microvascular complications in type 2 diabetes: the Japan Diabetes Complications Study/the Japanese Elderly Diabetes Intervention Trial risk engine
.
Diabetes Care
2013
;
36
:
1193
1199
Readers may use this article as long as the work is properly cited, the use is educational and not for profit, and the work is not altered. More information is available at https://www.diabetesjournals.org/content/license.

Supplementary data