Prediction of Coronary Heart Disease Risk in a General, Pre-Diabetic, and Diabetic Population During 10 Years of Follow-up: Accuracy of the Framingham, SCORE, and UKPDS Risk Functions: The Hoorn Study

The Hoorn Study is a population-based cohort study of glucose metabolism of 2,484 Dutch Caucasian men and women, aged 50–75 years at study initiation in 1989. The study design and population have been described in detail elsewhere (16).

For the current study, we included participants with NGT (n = 1,791), participants with intermediate hyperglycemia (n = 423), and participants with type 2 diabetes (n = 255) at baseline. After exclusion of individuals with a previous history of cardiovascular disease at baseline (NGT n = 294, intermediate hyperglycemia n = 99, and diabetes n = 77) and of individuals with missing values for any of the predictor variables (NGT n = 8, intermediate hyperglycemia n = 3, and diabetes n = 10) or outcome variables (NGT n = 364, intermediate hyperglycemia n = 89, and diabetes n = 43), 1,125 individuals with NGT, 232 individuals with intermediate hyperglycemia, and 125 individuals with diabetes remained for the current analyses. All participants provided written informed consent. Ethics approval for the study was obtained from the Ethics Review Committee of the VU University Medical Center Amsterdam.

According to baseline levels of fasting plasma glucose and 2 h after an oral glucose tolerance test, participants were classified into NGT, intermediate hyperglycemia, or diabetes groups, which were defined according to the 2006 criteria of the World Health Organization (17). Individuals who were already being treated for diabetes with insulin, oral hypoglycemic agents, or a physician-prescribed diet were categorized as known diabetic subjects, irrespective of their glucose levels.

Weight and height were measured in barefoot participants wearing light clothing. BMI was calculated as weight in kilograms divided by the square of height in meters. Blood pressure was measured in a sitting position after 5 min of rest using a random-zero sphygmomanometer (Hawksley-Gelman, Lancing, Sussex, U.K.). Serum total and HDL cholesterol levels were measured using an enzymatic technique (Boehringer-Mannheim, Mannheim, Germany) as described elsewhere (18). A1C was determined by ion-exchange high-performance liquid chromatography with a Modular Diabetes Monitoring System (Bio-Rad, Veenendaal, the Netherlands). Information about cigarette smoking, diabetes duration in those with a diagnosis of diabetes, ethnicity, medication use, history of cardiovascular diseases, and family history of myocardial infarction was assessed by self-administered questionnaires.

Primary end points defined for this study were CHD events that had occurred until 1 January 2000. Information on fatal and nonfatal CHD events was classified using medical records of general practitioners and the local hospital. Morbidity and fatal events were coded according to the ICD-9. CHD was defined as fatal and nonfatal ischemic heart disease and sudden death (ICD-9 codes 410–414, 427.4, 427.5, and 798).

Baseline characteristics are presented as means ± SD, median (interquartile range) in case of a skewed distribution, or percentages according to glucose status group. Estimated 10-year first CHD risk was calculated for each participant using the Framingham (5) and UKPDS (7) algorithms. Estimated 10-year risk of fatal CHD was calculated by means of the SCORE risk function developed for the low-risk region of Europe. To estimate the risk in the diabetic subgroup, the predicted risk by the SCORE algorithm was multiplied by 2 for men and by 4 for women, as recommended by Conroy et al. (6).

The predictive accuracy of the three risk functions was estimated using calibration (the ability to predict the number of observed events during follow-up) and discrimination appraisals (the ability to distinguish between those who experience a CHD event during follow-up from those who do not). These two approaches have been described in detail previously (19).

Calibration of the model was visually checked by plotting the predicted probabilities estimated by the prediction models against the observed proportion of first CHD events (Framingham and UKPDS) and fatal CHD events (SCORE). Participants were grouped into quintiles of predicted CHD risk within 10 years of follow-up. Plots were created for the three glucose status subgroups. The discriminatory ability of the models was assessed by calculating the area under the receiver operating characteristic curve (AUROC) using the 10-year risk estimates predicted by the three models. The discriminatory power of models is graded as low for an AUROC between 0.5 and 0.7, moderate between 0.7 and 0.9, and high if >0.9 (20).

Because the UKPDS risk function is derived from a cohort of patients with newly diagnosed diabetes and the Framingham and SCORE functions do not include information about diabetes duration, sensitivity analyses were carried out to test the discriminatory ability of the risk algorithms in predicting first or fatal CHD event in individuals with screening-detected diabetes only. Furthermore, because family history of myocardial infarction is a strong predictor of the incidence of CHD and is part of usual risk assessment in general practice (21), we tested whether the addition of family history of myocardial infarction to the prediction formulas improved risk estimation of the first CHD event. Improvements in AUROCs and reclassification improvements were examined and tested for significance using methods described previously (22,23). All statistical analyses were performed using SPSS 15.0 for Windows (SPSS, Chicago, IL).

Table 1 presents baseline and end point characteristics stratified by glucose status subgroup. Antihypertensive and lipid-lowering medication use was highest among diabetic subjects. A total of 197 first CHD events, of which 43 were fatal, were observed in this population. A higher percentage of first CHD events occurred in participants with diabetes compared with participants categorized in the NGT or intermediate hyperglycemia subgroup.

Figure 1 depicts the calibration plots for the prediction of first and fatal CHD events. The Framingham and UKPDS prediction models showed an overestimation of the risk of the first CHD event, mainly in the upper quintiles of predicted risk in the NGT, the intermediate hyperglycemia, and the diabetes subgroup. The estimation of fatal CHD events by the SCORE algorithm was fair in both the NGT and intermediate hyperglycemia group but less precise in the diabetic group.

The discriminatory results from the three models, applied to the Hoorn subpopulations, are presented in Table 2, showing the AUROCs and 95% CIs. The Framingham prediction formula showed the lowest AUROCs among all glucose status groups in estimation of first CHD risk. The SCORE risk function showed the highest area under the curve when fatal CHD risk in the NGT subgroup (AUROC 0.79 [95% CI 0.70–0.87]) and in the diabetic subgroup (0.74 [0.56–0.93]) was estimated. For the intermediate hyperglycemia group, moderate discriminatory ability (AUROC >0.70) was only seen for Framingham (0.76 [0.65–0.88]) and UKPDS (0.84 [0.74 – 0.94]) for fatal CHD risk. In the diabetic subgroup, all prediction models had low discriminatory ability for estimating the risk of first CHD event, whereas SCORE and UKPDS had moderate ability to identify individuals with a high risk for a fatal CHD event. The UKPDS risk model, derived from a diabetic population, showed a slightly higher area under the curve in a population with NGT (0.71 [0.66–0.75]) and in the population with intermediate hyperglycemia (0.70 [0.60–0.80]) compared with the diabetic population (0.66 [0.56–0.78]). Overall, all prediction models showed better discrimination when a fatal CHD event was used as the predicted outcome.

When the discriminatory ability of the three risk functions to estimate risk of a first CHD event in individuals with screening-detected diabetes was analyzed, all risk functions showed higher areas under the curve in this group compared with those in the group in which individuals with screening-detected diabetes and known diabetes were combined (Framingham 0.74 [95% CI 0.59–0.89], SCORE 0.79 [0.66–0.92], and UKPDS 0.75 [0.63–0.87]). Similar results were seen for estimations of a fatal CHD event (Framingham 0.73 [0.50–0.95], SCORE 0.82 [0.67–0.97], and UKPDS 0.83 [0.68–0.97]). The addition of family history of myocardial infarction slightly improved most risk algorithms in prediction of the risk of a first CHD event although the changes were not statistically significant.

In this study, we compared the performance of two risk functions designed for the general population and one diabetes-specific risk function in predicting nonfatal and/or fatal CHD risk in a general, pre-diabetic, and diabetic population during 10 years of follow-up. With respect to the agreement between the predicted and observed estimates, the Framingham and UKPDS risk function overall overestimated the actual observed CHD incidence rate in the three subgroups. Regarding the risk of first CHD, the Framingham algorithm had low ability to discriminate in all subgroups except for the group consisting of patients with screening-detected diabetes in whom the discriminatory ability was moderate. The UKPDS function had moderate ability to identify those with high risk for a first CHD event in the NGT and intermediate hyperglycemia groups and in patients with screening-detected diabetes but low ability in the group in which patients with screening-detected and known diabetes were combined. The SCORE algorithm for the prediction of fatal CHD has a moderate ability in all subgroups. Although the Framingham and UKPDS risk functions were designed to estimate first CHD in the general population and the diabetic population, respectively, both functions performed better in estimating fatal CHD than the SCORE risk function for fatal CHD with the highest discriminatory ability being observed for the UKPDS risk function in the intermediate hyperglycemia group.

The results of our study do not support the findings in previous studies in which the CHD risk in a population with diabetes was underestimated by the Framingham (10,14) and UKPDS risk functions (12). Previous research also showed an overestimation of CHD risk predicted by the UKPDS risk function during 5 years of follow-up (24), the SCORE risk function during 10 years of follow-up (11), and the Framingham risk function in European populations (8,9). Recently, the performance of the Framingham and UKPDS risk functions for estimation of cardiovascular disease risk has been tested, and both functions were found to overestimate the cardiovascular disease risk and to be moderately effective in a normoglycemic, pre-diabetic, and diabetic population (25). These results are comparable to the predictive accuracy of the CHD risk functions in our study despite the different criteria used to define the three glucose status groups.

Some possible reasons for the overestimation of CHD risk by the Framingham and UKPDS prediction models in our study population can be suggested. Prediction of the risk of CHD, calculated by the Framingham function, might result in less precise CHD estimates in a population well treated for cardiovascular risk factors, as is the case with the Hoorn population. This phenomenon may also contribute to the better prediction in patients with newly diagnosed diabetes compared with those with known diabetes. In addition, it is suggested that risk estimation might be more accurate in individuals with fewer risk factors (4). Indeed, in the current study, a larger overestimation of the actual observed CHD events was seen in individuals in the higher quintiles of estimated CHD risk. Regarding the SCORE risk function for estimating fatal CHD events, the estimation could have been less precise because of the small number of observed fatal CHD events in the current study. In addition, participants with missing information on CHD events were excluded, and a selection bias could have occurred. The population used for the current study consists of Caucasian men and women. Because performance of the risk functions might differ between different ethnic groups, the results of our study cannot be immediately generalized to populations of other ethnic origin. Furthermore, the UKPDS risk function was based on patients with newly diagnosed diabetes, and our results suggested better performance of the risk algorithms when they were applied solely to patients with newly detected diabetes. The SCORE and Framingham risk functions also showed better discriminatory ability in patients with screening-detected diabetes. Values of risk factors used for the development of the UKPDS prediction model were based on measurements taken at 1 or 2 years after diagnosis of diabetes. At that time treatment has changed, which resulted in altered risk profiles for most patients. In our diabetic subgroup, 68% of the participants had the diagnosis of diabetes at baseline and risk factor levels at the baseline measurement were used for all analyses. The different time of measurement could be a reason for the overestimation of the risk of first CHD calculated by the UKPDS risk engine.

In summary, we found that the use of the Framingham function in the prediction of first CHD event in all participants in the Hoorn cohort was likely to overestimate an individual's absolute CHD risk. From a clinical perspective, this overestimation may lead to improper targeting of preventive strategies because of the large number of patients with false-positive results who are identified for treatment. From a health policy point of view, this may cause incorrect allocation of health resources. The equation might be used to estimate potential relative reduction in CHD risks. To aid in CHD prevention, application of the SCORE risk function in diabetic patients and application of the SCORE and UKPDS risk functions in individuals with normal glucose metabolism or intermediate hyperglycemia might prove useful in the absence of a more valid tool.

The Hoorn Study has been made possible by Vrije Universiteit Amsterdam, the VU University Medical Center, the Dutch Diabetes Research Foundation, the Dutch Organization for Scientific Research, the Netherlands Heart Foundation, and the Health Research and Development Council of the Netherlands. A.A.W.A.v.d.H. was supported by the Netherlands Organization for Health Research and Development.

No potential conflicts of interest relevant to this article were reported.