Derivation and External Validation of a Clinical Model to Predict Heart Failure Onset in Patients With Incident Diabetes

Over the last four decades, the number of adults with diabetes almost quadrupled worldwide (1). Having diabetes more than doubles an individual’s risk of developing cardiovascular disease (CVD) (2), which is a leading cause of mortality (3) accounting for half of all deaths in patients with diabetes (4). Heart failure (HF) is an important sequela of diabetes through accelerated atherosclerosis and other direct cellular mechanisms (5) and is increasingly being recognized for its role in the cardiovascular morbidity and mortality seen in this population (6). Data from clinical trials suggest that sodium–glucose cotransporter 2 inhibitors (SGLT-2i) may reduce the likelihood of incident HF in patients with diabetes (7). The ability to accurately identify individuals at risk for developing HF provides an opportunity for personalized preventative therapy, potentially reducing the CV disease burden in patients with diabetes.

Although several risk models have been developed to predict the onset of HF in patients with diabetes, the majority are based on patients with prevalent instead of new-onset diabetes (8), potentially missing the maximal window of opportunity for personalized prevention. Additionally, existing risk scores are based on clinical trial or aggregated cohort study data that lack real-world representation. They also fail to address the fundamental differences in HF risk, risk factors, and outcomes in women and men. Women with type 2 diabetes have been found to be at higher risk of developing HF than men. A meta-analysis of 14 studies encompassing >12 million individuals found that diabetes conferred 38% excess risk of HF, as well as a greater excess risk of all-cause and cardiovascular death in women (9). These data emphasize the need for sex-specific approaches to risk stratification and management of patients with diabetes (10).

Given this need, we used population-based administrative data in Ontario, Canada to derive clinical risk models to predict the onset of HF in adult women and men at the time of diabetes diagnosis and externally validated these models on a concurrent cohort of patients in the U.K.

Included in this retrospective cohort study were adult patients ≥18 years of age who were newly diagnosed with diabetes. Those who were ≥105 years of age, were long-term care residents, or were dialysis-dependent at the time of diabetes diagnosis were excluded. Patients with no HbA_1c within 60 days before and 30 days after diabetes diagnosis and those already diagnosed with HF at the time of diabetes diagnosis were also excluded.

The Ontario cohort consisted of all patients with incident diabetes diagnosed between 1 April 2009 and 31 March 2018. Ontario is the most populous province in Canada with 13 million residents and one of the most ethnically diverse jurisdictions in the world. We used population-level administrative health care databases that are held securely in coded form and analyzed at the Institute for Clinical Evaluation Sciences (ICES). ICES is an independent, nonprofit research institute for which legal status under Ontario’s health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement.

Incident cases of diabetes were identified using a validated algorithm, based on one inpatient or two outpatient physician service claims for diabetes within 2 years. This algorithm was shown to have 86% sensitivity and 97% specificity for identifying the onset of any nongestational diabetes when validated in primary care patient records (11). We linked these records with the Registered Persons Database (demographic and vital statistics), the Canadian Institute for Health Information (CIHI) Discharge Abstract Database (hospitalizations and comorbidities), and the Same Day Surgery database (comorbidities). Physician service claims data were obtained from the Ontario Health Insurance Plan database and laboratory values from the Ontario Laboratory Information System. These databases have been validated for many outcomes, exposures, and comorbidities (11,12).

We estimated socioeconomic status based on patients’ neighborhood median income in the Canadian census and determined rural versus urban residence using Statistics Canada definitions (13). We identified hypertension (12), asthma, and chronic obstructive pulmonary disease (COPD) (14) using validated algorithms and other comorbidities using the Discharge Abstract Database, Same Day Surgery, and Ontario Health Insurance Plan databases based on ICD-10 Canada codes on patient encounters within 5 years of diabetes diagnosis, using previously described methods (15,16).

The external validation cohort consisted of all patients aged ≥35 years with a diagnostic code for incident type 2 diabetes, who met the selection criteria between 1 March 2007 and 31 March 2017, and registered with general practices in the U.K. eligible for linkage to external data sets. The cohort was derived from the U.K. Clinical Practice Research Datalink (CPRD) Global Initiative for Chronic Obstructive Lung Disease (GOLD) database, which is one of the world’s largest electronic medical records (EMR) providing anonymized patient-level data and deemed as representative of the U.K. population (17).

The CPRD GOLD primary care records contain sociodemographic, clinical, therapy, laboratory, and referral information from 1987 onward. We linked the CPRD records of the eligible type 2 diabetes cohort to Hospital Episode Statistics, which contains data on all hospital admissions, the Office for National Statistics mortality data, and the index of multiple deprivation (IMD) quintiles 2015 as a measure for socioeconomic status. IMD is recorded at the patient’s residential postcode level and represents a score calculated as the weighted sum of seven deprivation domains, of which income and employment are the highest contributing domains.

Using an algorithm that categorizes patients’ alcohol consumption status into six categories based on general practice codes (18), alcoholism refers to patients deemed to be heavy drinkers. Mood disorder definition covered conditions including adjustment disorders, bipolar affective disorders, manic episodes, depersonalization-derealization syndrome, depressive episodes, dissociative amnesia, dissociative stupor, and anxiety disorders.

The primary outcome was incident HF. In the Ontario cohort, this was identified by a validated algorithm with 85% sensitivity and 97% specificity based on one inpatient or two outpatient billing claims for HF within 1 year (19). HF was identified from primary care and linked secondary care Hospital Episode Statistics records in the U.K. cohort using Read and ICD-10 codes, respectively. The validity of cardiovascular diagnoses in CPRD is recognized (20).

Continuous variables were compared with a two-sample t test or Wilcoxon rank sum test where appropriate. Categorical variables were compared with a χ² test. Outcomes were assessed through 31 March 2019. In Ontario, patients were censored when they died or were no longer eligible for Ontario health insurance. In the U.K., patients were censored when they died or left the CPRD-contributing practice.

Model development was based on Ontario data and stratified by sex. We split the female and male cohorts by random selection such that 70% of each cohort was used for derivation and 30% for internal validation (21). The prediction of HF was accomplished using Fine and Gray subdistribution hazard models within a competing risk framework (22). Candidate variables from Table 1 were selected based on Bayesian information criteria, using a backward stepwise elimination model with death as the competing event (23).

Rurality and socioeconomic status were missing in <0.1% of patients, glomerular filtration rate (GFR) in 21,078 (6.1%), and hemoglobin in 28,744 (8.3%). We imputed missing values once within the SAS “proc MI” framework, where they were predicted drawing on all candidate covariates using predictive mean matching for continuous variables and logistic regression for categorical variables (24). We examined the association between each continuous variable with the outcome using cubic spline analyses with three knots at percentiles 10, 50, and 90. As the linearity assumption held for all variables, they were entered into the model as continuous values. We validated the model on the remaining 30% of the cohort. We reported subhazard ratios, 95% CIs, and P values for final covariates in each model.

We evaluated model discrimination using the C-statistic and estimated 95% CIs using 200 bootstraps. We assessed calibration using Brier scores (25) and time-dependent plots of observed versus predicted HF incidence rates within deciles of predicted risk in the validation cohort.

We conducted two sensitivity analyses to assess the robustness of our models in clinically relevant settings. First, as SGLT-2i is the recommended first-line therapy for patients with diabetes and CVD, we evaluated the performance of our models in patients without CVD. Specifically, per European and American Diabetes Association guidelines (26,27), we excluded those with a history of CVD, including ischemic heart disease (IHD), cerebrovascular disease, and peripheral arterial disease. Second, we evaluated the performance of our models in predicting incident HF hospitalization.

Analyses were performed using SAS 9.4 (SAS Institute, Cary, NC) and R studio version 3.5.1 in the Ontario cohort and Stata 16.1 (College Station, TX) in the U.K. cohort, with statistical significance defined by a two-sided P value <0.05.

In Ontario, the use of data was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board. In the U.K., the study was approved by the Independent Scientific Advisory Committee for the Medicines and Healthcare Products Regulatory Agency Database Research (protocol number 17_168). Generic ethnical approval for observational research using CPRD with approval from the Independent Scientific Advisory Committee has been granted by Health Research Authority Research Ethics Committee (Ease Midlands–Derby, REC reference number 05/MRE04/87).

For Ontario, the data set from this study is held securely in coded form at ICES. While legal data sharing agreements between ICES and data providers (e.g., health care organizations and government) prohibit ICES from making the data set publicly available, access may be granted to those who meet prespecified criteria for confidential access, available at https://www.ices.on.ca/DAS (e-mail: [email protected]). The full data set creation plan and underlying analytic code are available from the authors upon request, understanding that the computer programs may rely upon coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification. In the U.K., access to data can be requested via application to the CPRD.

The Ontario cohort was used for derivation and internal validation. No patients were lost to follow-up. Median follow-up for the 348,027 Ontarians with incident diabetes (45.0% women) was 5 years (interquartile range [IQR] 3–7), and maximum follow-up was 9 years. In the derivation cohorts, 4,027 (3.7%) among 109,600 women and 5,803 (4.3%) among 134,018 men developed HF during the follow-up period. In the internal validation cohorts, 1,742 (3.7%) among 46,972 women and 2,469 (4.3%) among 57,437 men developed HF.

The external validation cohort comprised 54,483 U.K. residents with incident type 2 diabetes (45.3% women). No patients were lost to follow-up. Median follow-up duration was 5 years (IQR 3–7). Among 24,664 women, 1,107 (4.5%) developed HF. Among 29,819 men, 1,494 (5.0%) developed HF (Supplementary Table 1).

The baseline characteristics of the Ontario and U.K. cohorts were similar within each sex, with the exception that the Ontario patients were younger, less likely to have atrial fibrillation (AF), IHD, cerebrovascular disease, peripheral arterial disease, hypothyroidism, alcoholism, chronic renal disease, and primary malignancy. Ontarians were, however, more likely to have long-standing hypertension, valvular heart disease, or COPD or to undergo cardiac testing, including invasive or computed tomography angiography, nuclear medicine test, cardiac positron emission test, or cardiac stress test (persantine, dobutamine, electrocardiogram, or stress echocardiography) (Table 1).

The multivariable risk factors of HF in women were age, rurality, duration of hypertension, valvular heart disease, IHD, AF, history of cardiac testing, COPD, alcoholism, baseline hemoglobin, HbA_1c, GFR, and mood disorder (Table 2).

The performance of the models on derivation and internal and external validation is summarized in Table 3. In the internal validation data set, the C-statistics at 1, 5, and 9 years were 0.79 (95% CI 0.77–0.82), 0.79 (0.78–0.81), and 0.77 (0.76–0.79); and Brier scores were 0.007 (95% CI 0.006–0.008), 0.034 (0.032–0.035), and 0.066 (0.062–0.069), respectively. In the external validation data set, the C-statistics at 1, 5, and 9 years were 0.81 (0.79–0.84), 0.79 (0.77–0.80), and 0.78 (0.76–0.79); and Brier scores were 0.010 (0.009–0.011), 0.038 (0.035–0.041), and 0.067 (0.062–0.071), respectively, indicating excellent calibration. Supplementary Figs. 1–3 show the area under the receiver operating characteristic curves, and Fig. 1 shows the calibration plots of observed versus expected rates of HF at 1, 5, and 9 years after diabetes diagnosis in women according to each decile of risk. The model calibrated well in all but the highest risk decile in the external validation cohort, in which the model tended to overpredict.

The multivariable risk factors of HF in men were age, rurality, income quintile, duration of hypertension, valvular heart disease, IHD, history of coronary artery bypass grafting, AF, history of cardiac testing, COPD, liver disease, hemoglobin, HbA_1c, and GFR (Table 2).

In the internal validation data set, the C-statistics at 1, 5, and 9 years were 0.75 (0.73–0.78), 0.78 (0.77–0.79), and 0.76 (0.75–0.77); and Brier scores were 0.008 (0.007–0.009), 0.039 (0.037–0.041), and 0.073 (0.070–0.076), respectively. In the external validation data set, the C-statistics at 1, 5, and 9 years were 0.78 (0.75 to 0.80), 0.77 (0.76 to 0.79), and 0.77 (0.75 to 0.79); and Brier scores were 0.011 (0.010–0.012), 0.044 (0.042–0.047), and 0.072 (0.068–0.076), respectively, indicating excellent calibration (Table 3). Supplementary Figs. 1–3 show the area under the receiver operating characteristic curves, and Fig. 1 shows the calibration plot of observed versus expected rates of HF at 1, 5, and 9 years after diabetes diagnosis in men according to each decile of risk. The model calibrated well in all except the highest risk decile in the external validation cohort tended, in which the model tended to overpredict.

For the female-specific model, the C-statistics at 1, 5, and 9 years were 0.82, 0.81, and 0.80, and Brier scores were 0.0010, 0.0082, and 0.022, respectively, in the Ontario cohort. The C-statistics were 0.81, 0.78, and 0.77, and Brier scores were 0.0065, 0.028, and 0.049, respectively, in the U.K. cohort.

For the male-specific model, the C-statistics were 0.78, 0.78, and 0.77, and Brier scores were 0.0011, 0.0086, and 0.022, respectively, in the Ontario cohort. The C-statistics were 0.74, 0.76, and 0.75, and Brier scores were 0.0067, 0.029, and 0.051, respectively, in the U.K. cohort.

For the female-specific model, the C-statistics at 1, 5, and 9 years were 0.83, 0.82, and 0.81, and Brier scores were 0.0013, 0.0099, and 0.025, respectively, in the Ontario cohort. The C-statistics were 0.81, 0.79, and 0.77, and Brier scores were 0.0098, 0.038, and 0.067, respectively, in the U.K. cohort.

For the male-specific model, the C-statistics were 0.80, 0.80, and 0.78, and the Brier scores were 0.0014, 0.011, and 0.027, respectively, in the Ontario cohort. The C-statistics were 0.78, 0.77, and 0.76, and Brier scores were 0.011, 0.044, and 0.072, respectively, in the U.K. cohort.

Our sex-specific models are together termed the Predicting Heart Failure in Diabetes (PARFAIT) models. These models have been adapted into risk calculators and provided in the Supplementary Material.

We derived and validated sex-specific models to predict incident HF in adults with new-onset diabetes. We observed that 3.7–4.5% of women and 4.3–5% of men with diabetes developed HF over the study period and that age, duration of hypertension, GFR, hemoglobin, HbA_1c, and prevalent CVD are risk factors of HF common to both sexes. Specific to women, mood disorder and alcoholism are additional HF risk factors, while income and liver disease are male-specific risk factors. The performance of our models was robust over a 9-year follow-up. We have enclosed automated risk calculators to make these models readily applicable in clinical settings.

The prevalence of diabetes is growing globally at a rapid pace. A previous U.K. population-based study reported that patients with type 2 diabetes are more than twice as likely to have HF as their age-sex-practice–matched comparators without diabetes (men, odds ratio 2.12 [95% CI 1.76–2.54]; women, odds ratio 2.27 [1.81–2.85]) (28). Given HF’s role in the development of disability (29) and other adverse long-term outcomes (16,30), the ability to predict HF risk will inform timely and personalized preventative therapy. Lifestyle modification and other interventions, such as SGLT-2i, have cardioprotective benefits, and the latter is associated with ∼25% reduction in HF hospitalizations and cardiovascular death (7,31). Our work describes new high-risk features in the development of HF and highlights the importance of sex-specific models in predicting future risk, particularly when risk factors may vary by sex. Additionally, our models exhibited excellent performance in predicting severe HF requiring hospitalization, particularly among patients for whom prophylactic SGLT-2i is not routinely recommended by guidelines. These features may play an important role in further reducing population-level risk of HF.

Many existing risk scores were derived and validated in smaller cohorts and lack robust external validation, which effectively limits their applicability in the real world (32–36). A number of these models were based on older clinical trial data and predicted a variety of HF-related outcomes instead of HF onset: the WATCH-DM risk score was derived using machine-learning methods based on data from 1999 to 2009 and predicted HF in patients with prevalent type 2 diabetes (C-statistic 0.77 on internal validation) (36); the Thrombolysis in Myocardial Infarction (TIMI) Risk Score for HF in Diabetes (TRS-HFDM) predicted HF hospitalization (C-statistic 0.78) (32); Pfister et al. (35) calculated the risk for HF in people with advanced type 2 diabetes complicated by macrovascular disease (C-statistic 0.75); and the UK Prospective Diabetes Study Outcomes Model (UKPDS-OM) estimated the absolute probability of first occurrence of seven major diabetes-related complications, HF being one of them (33). Pandey et al. (34) used data from three cohort studies to predict HF among patients with prediabetes or prevalent diabetes using biomarker-based risk score (C-statistic 0.74). A recent HF hospitalization risk model was based on EMR data of 54,452 predominately Caucasian patients with incident or prevalent type 2 diabetes from a U.S.-based single-payer system (C-statistic 0.782) (37). In contrast, our models were derived from an ethnically diverse, contemporary population of >250,000 patients within a universal health care system. Our models were validated externally across continents demonstrating excellent performance at all time points. In addition to their demonstrated applicability around the world, these models uniquely apply at the onset of diabetes thus affords a larger window of opportunity for HF prevention (8,35).

Notably, most published HF risk models in patients with diabetes are non–sex-specific, in spite of known sex differences in cardiovascular risk attributable to diabetes (9,10) and the differences in HF risk factors and outcomes in women and men (16,30,38). The only exceptions to this are models derived by Hippsley-Cox and Coupland (39), which were not specific to patients with incident diabetes, incorporated the exact same predictors in both sexes, and lacked external validation. These models had similar performance to ours (C-statistics 0.769 in men and 0.783 in women).

The availability of long-term follow-up data are essential to the clinical applicability of models that predict the onset of chronic disease. Our follow-up duration exceeds most similar studies. The derivation cohort by Pfister et al. (35) had a mean follow-up of 34.5 months. A study on the posttrial monitoring data for the UKPDS-OM risk score found that the HF prediction model performed well in the first 3 years but overpredicted at 10 years (40). Our models demonstrated consistent performance throughout 9-year follow-up and are based on routinely collected data, which demonstrates generalizability to jurisdictions with and without established EMR systems.

A number of risk factors have been reported in association with the future risk of HF in patients with diabetes, including glycemic control, CVD, or cardiovascular risk factors (37), renal function (35), and sociodemographic factors such as age and income (39,41). Our models included these variables and also highlighted other high-risk features such as hemoglobin, COPD, and alcoholism. Additionally, our models are based on simple data elements that are readily available in the primary care setting, in which diabetes is most frequently diagnosed and managed. This makes our models widely applicable in the community. Our report of sex-specific risk factors is important and is, to our knowledge, a first step toward personalized preventative medicine.

Our study has several limitations. First, our derivation cohort contained a mixed population with type 1 and 2 diabetes. However, these models performed well in an external validation cohort of exclusively patients with type 2 diabetes. Second, our data sources do not capture diabetes subtype, nor routinely capture measures of physical activity and other lifestyle factors that may have important roles in the development of incident HF. Third, as drug coverage is only publicly funded for Ontarians >65 years of age, we were unable to incorporate medications into the development of our model.

We developed and externally validated sex-specific risk models to predict long-term HF risk in patients with new-onset diabetes to maximize the window of opportunity for preventative therapy. Our models demonstrated robust performance over a 9-year follow-up. Our identification of sex-specific risk factors, the ability of our models to additionally predict severe HF requiring hospitalization, as well as among patients for whom prophylactic SGLT-2i is not routinely prescribed by current guidelines, represent an important step toward personalized lifestyle and pharmacologic prevention to potentially allow millions of patients with diabetes to live longer and better.

Acknowledgments. The authors acknowledge the usage of data compiled and provided by the CIHI. These data sets were linked using unique encoded identifiers and analyzed at ICES.

The analyses, conclusions, opinions, and statements expressed in the manuscript are those of the authors and do not necessarily reflect those of the above agencies.

Funding. This study was supported by the Canadian Institutes of Health Research. L.Y.S. was named National New Investigator by the Heart and Stroke Foundation of Canada and is supported by a Tier 2 Clinical Research Chair in Big Data and Cardiovascular Outcomes at the University of Ottawa. D.S.L. is supported by a Mid-Career Investigator Award from the Heart and Stroke Foundation. K.T. received a research scholar award from the Department of Family and Community Medicine at the University of Toronto. This study was also supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care. Parts of this material are based on data and information compiled and provided by the CIHI. The authors acknowledge that the clinical registry data used in this analysis are from participating hospitals through CorHealth Ontario, which serves as an advisory body to the MOH, is funded by the MOH, and is dedicated to improving the quality, efficiency, access, and equity in the delivery of the continuum of adult cardiac and stroke care in Ontario, Canada. The U.K. cohort study was funded by the National Institute for Health Research School for Primary Care Research.

The analyses, conclusions, opinions and statements expressed in this study are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. L.Y.S. was responsible for conception and design. L.Y.S., S.S.Z., and A.B.E. were responsible for data acquisition and analysis. L.Y.S., S.S.Z., A.B.E., P.P.L., D.S.L., K.T., S.W.T., E.K., and M.A.M. were responsible for interpretation of data. L.Y.S. and S.S.Z. drafted the manuscript. L.Y.S., S.S.Z., A.B.E., P.P.L., D.S.L., K.T., S.W.T., E.K., and M.A.M. were responsible for critical revision and final approval. L.Y.S. is the guarantor of this work and, as such, had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.