The Performance of a Risk Score in Predicting Undiagnosed Hyperglycemia

The development and evaluation of the risk score have been previously described (10). In brief, a predictive score was produced using stepwise logistic regression on data from a study of incident cases of clinically diagnosed type 2 diabetes (the Wessex study) (12) and a study of prevalent cases of detected diabetes detected through screening (the Ely study) (13). The variables used in the risk score were age, sex, prescribed medication (steroid and antihypertensive), smoking, family history, and BMI. The parameters for the risk score are reproduced in the appendix. The score was evaluated in an independent, randomly selected population sample from the Ely study who underwent oral glucose tolerance testing. The sensitivity, specificity, and area under the receiver-operating characteristic (ROC) curve were 82, 62, and 79.5 (95% CI 72.6–86.3), respectively.

The design of the EPIC-Norfolk study, including recruitment and measures, has been reported in full elsewhere (14). This study is a multicenter prospective cohort study examining the relation between dietary and other lifestyle factors and the risk of developing various chronic conditions, including cancer. Participants completed a detailed health and lifestyle questionnaire, including questions on current medication, smoking habits, and family history of diabetes. After the questionnaire, the subjects were invited for a health check-up at a general practice facility. The factors assessed at the check-up included anthropometric measures and blood pressure; blood samples were also taken for later testing.

This study used data from the 6,567 people from the EPIC-Norfolk cohort who were between ages 39 and 78 years, who had no known diabetes, and for whom complete data were available on HbA_1c and risk score variables by July 1998. HbA_1c testing was started in November 1995, and the population having complete data by July 1998 formed the cohort for this analysis. The 535 subjects with known diabetes or incomplete risk score data were excluded from the study. Excluded individuals did not differ significantly from those included for any risk score variables. The cohort was representative of the general population of England and Wales for age and sex distribution, anthropometric variables, blood pressure, and serum lipids (14); 99.7% of the cohort was Caucasian. Undiagnosed hyperglycemia was defined by an HbA_1c ≥7%. The HbA_1c assays were carried out using a Diabetes Control and Complications Trial aligned assay on high-performance liquid chromatography (Biorad Diomat). The intra-assay coefficient of variation was 3.6% at the lower end of the range (mean 4.94%) and 3.0% at the upper end (mean 9.76%).

GHb does not feature in either the World Health Organization or the American Diabetes Association definition of diabetes, largely because of issues of standardization among assays and the lack of availability of assays in certain areas of the world. Within a cohort, however, it is a reasonable measure of hyperglycemia. The cut-off of 7% was chosen as the major definition of hyperglycemia because at this level the future risk of the specific microvascular complications of diabetes rises steeply, and a recent meta-analysis has shown that only 4% of those above this cut-off will be misclassified as having diabetes (15). A recent comparison by Hanson et al. (16) of fasting and 2-h post-challenge glucose concentrations and GHb showed that all three tests were equally predictive of future retinopathy, and that any could be used as the basis for the definition of diabetes.

The choice of an optimal threshold for defining diabetes on the basis of GHb is a highly contentious issue. An alternative approach to resolving this, other than studying the relation of GHb and future risk, is to examine population distributions for bimodality. The study by Hanson et al. (16) estimated that the threshold of GHb associated with the least overlap between the two peaks in their bimodal population and also associated with a significantly higher risk of retinopathy was 7.8%. A study by Engelgau et al. (17) using similar methods found an optimum diagnostic threshold for GHb of 6.7%. The Japanese Diabetes Society has recommended the use of GHb at a threshold of 6.5% as a supportive tool for diagnosis (18). Because of these varying recommended thresholds, this study assessed the performance of the risk score against a range of thresholds of GHb, defining hyperglycemia as HbA_1c ≥6.0, 6.5, and 7%.

The diabetes risk score was calculated for the entire sample. Sensitivity, specificity, predictive values, and likelihood ratios for detection of undiagnosed hyperglycemia were calculated for cut-offs of the risk score between 0 and 1. ROC curves were plotted, and area under the curve (AUC) values and their CIs were estimated using the nonparametric method described by Hanley and McNeil (19). The Statistical Package for the Social Sciences Program, Version 8.0 (SPSS), was used for analysis. CIs for the performance characteristics of the risk score were calculated using Confidence Interval Analysis software, version 1.0 (20).

The performance of the risk score was then estimated with data excluded for different variables in turn. First, family history was omitted, then family history and smoking history, then family history and BMI, and finally BMI, family history, and smoking history. Of the variables used by the risk score, BMI, family history, and smoking history were considered the most likely to be unavailable in routine clinical data.

The prediction model (PM1) developed by Baan et al. (11) in Holland was also applied to the entire EPIC sample; its performance in predicting an HbA_1c ≥7% was compared with the risk score. The PM1 model is the only statistical risk model in the literature comparable with the Cambridge risk score, in that it uses routinely available risk factor data to predict undiagnosed diabetes. The other model developed by Baan et al. (PM2) showed a better performance in the original data (11), but it was not appropriate for comparison in this study, as it required information on regular bicycle use, which would not be routinely available or culturally appropriate in the U.K.

The characteristics of the 6,567 individuals in the EPIC sample are shown in Table 1. In all, 84 individuals (1.3%) had previously undiagnosed hyperglycemia, as defined by an HbA_1c ≥7%. Age, hip circumference, BMI, family history of diabetes, and smoking habits were all significantly associated (P < 0.01) with an HbA_1c ≥7%. The association of the risk score variables with previously undiagnosed hyperglycemia in the EPIC-Norfolk sample is shown in Fig. 1.

The performance of the risk score in the whole EPIC sample against a GHb ≥7% is shown in Table 2. For a specificity of 78%, sensitivity was 51% (CI 40–62). The AUC was 74.2% (69.5–79.0), slightly greater than the area for PM1 of Baan et al. (73.2%; 68.3–78.1). The ROC curves for the risk score and the PM1 in the whole sample are shown in Fig. 2.

When the diagnostic threshold for elevated GHb was lowered to 6.5%, the number of those with undiagnosed hyperglycemia rose to 203 (3.1%). The AUC values for the risk score and PM1 for this threshold were 71.8% (CI 68.4–75.2) and 70.7% (67.3–74.1), respectively. When the threshold for GHb was 6.0%, the number of those with undiagnosed hyperglycemia was 841 (12.8%). The AUC values for the risk score and PM1 were 65.7% (63.8–67.6) and 64.3% (61.7–66.5), respectively.

When family history data were excluded from the risk score, the AUC actually rose to 75.1% (CI 68.5–81.7), although it was not significantly different from that of the original score. With smoking and family history excluded, the AUC rose further to 75.2% (68.7–81.7). With BMI and family history missing, the AUC for a threshold of 7% dropped significantly to 68.9% (63.7–74.1), and with BMI, smoking history, and family history excluded, the AUC was 69.8% (64.0–75.6). The performance of the risk score with missing variables was not significantly different from that of the original score for different thresholds of GHb (6.0 and 6.5%).

When tested in the EPIC cohort, a large, independent, and representative sample of the general population, the risk score performed reasonably well at identifying individuals at high risk for having undiagnosed hyperglycemia. The AUC value, the most useful comparative measure of the performance of screening tests (19), for the risk score in the EPIC population was 74.2%. This was not significantly different from the AUC in the original evaluation, but it was slightly higher than the value for the closest comparable tool, the PM1 model of Baan et al. (11) for all thresholds of GHb that were tested.

Other groups, such as the American Diabetes Association (21), have developed questionnaires to combine risk factor data efficiently when identifying those at high risk of undiagnosed diabetes. The AUC for the risk score was higher than those reported for two risk factor questionnaires: the Herman et al. questionnaire (22) and the Ruige at al. questionnaire (23) in a sample of Dutch people aged 50–74 years (23). The risk score also has comparable performance (AUC) in detecting undiagnosed diabetes to some biochemical tests, including fasting capillary glucose (24), 1-h postprandial capillary glucose (25), random capillary glucose (25), and glycosuria testing (16). Although the questionnaires and biochemical tests were evaluated against the 2-h plasma glucose, a more widely accepted gold standard than the GHb used in this study, GHb has been shown to correlate well with the development of clinical outcomes characterizing diabetes, such as retinopathy (17,26). However, it has only recently been proposed as a diagnostic test for diabetes (17,27,28). There will inevitably be some variation in the individuals diagnosed with diabetes by glucose tolerance test and GHb. Nevertheless, this is unlikely to have had a large impact on the estimate of the performance of the risk score in the present cohort.

Given that the risk score uses existing patient information, it may have a useful role in stratifying a practice population so that only those at highest risk are offered diagnostic testing, rather than the whole population. This approach is likely to generate fewer anxiety-provoking false-positive results than population screening by questionnaire or urine test, as well as fewer costs, particularly if the score is automatically calculated by a practice computer. The benefits, costs, feasibility, and acceptance of such an approach in primary care are currently undergoing evaluation in a randomized clinical trial (29). This trial will also address the broader, unresolved question of whether the benefits of screening outweigh the costs (30).

The risk score performed well even in the absence of data on the variables that are less likely to be routinely recorded in general practice records, such as family history, with AUCs of 68.9–75.2%. Variables such as BMI and age contributed more to the predictive models than did, for example, family history. This suggests that the risk score may have utility in general practice, where some routine clinical information may be unavailable. Furthermore, BMI can be reliably calculated from self-reported height and weight when data are incomplete (31,32).

There were some minor differences between the methods of data collection for development and testing of the risk score and those used in the EPIC population. For example, clinical records of prescribed medications were used in the Wessex study rather than the self-reports used in the EPIC. Such a discrepancy is likely to have led to an underestimation of the performance of the risk score. Also, both the original Ely sample and the EPIC population were mostly Caucasian. Ethnic differences are known to affect the prevalence of type 2 diabetes, prevalence of risk factors, and the extent to which undiagnosed diabetes is associated with various risk factors (33,34,35). Consequently, the performance of the risk score and other similar methods may be different in areas with greater ethnic diversity; these methods should be tested in such populations.

In conclusion, a simple risk score incorporating data routinely collected in general practice is effective in detecting individuals at high risk of having undiagnosed hyperglycemia, as defined by elevated GHb, in a large and representative population. Such a score could form part of a strategy for reducing the current delay between development and recognition of type 2 diabetes and an alternative to either testing all those over age 45 years, as is recommended by the American Diabetes Association (7), or distributing and analyzing risk factor questionnaires.

The logistic model predicting risk of prevalent diabetes (10) is defined as

Probability of having type 2 diabetes

where α = −6.322, β₁x₁ = −0.879 (x₁ = 1 if female, 0 if male); β₂x₂ = 1.222 (x₂ = 1 if prescribed antihypertensive medication, 0 = if not); β₃x₃ = 2.191 (x₃ = 1 if prescribed steroids, 0 if not); β₄x₄ = 0.063 (x₄ is age in years); β₅x₅ = 0 if BMI <25, 0.699 if BMI ≥25 and <27.5, 1.97 if BMI ≥27.5 and <30, and 2.518 if BMI ≥30; β₆x₆ = 0 if the person has no first degree relative with diabetes, 0.728 if there is a parent or sibling with diabetes, or 0.753 if there is both a parent and sibling with diabetes; β₇x₇ = 0 for non-smokers, −0.218 for ex-smokers, and 0.855 for current smokers.

P.J.P. was supported by the National Health Service Research and Development Directorate (Eastern). N.J.W. is a Medical Research Council Clinician Scientist fellow. L.S. was supported by the Cambridge Commonwealth Trust. The cohort of EPIC-Norfolk is supported by grant funding from the Cancer Research Campaign, the Medical Research Council, the Stroke Association, the British Heart Foundation, the Department of Health, the Europe Against Cancer Programme Commission of the European Union, and the Ministry of Agriculture, Fisheries, and Food.

We thank all those involved in the EPIC study, but particularly Nick Day, Kay-Tee Khaw, Sheila Bingham, Ailsa Welch, Robert Luben, and Suzy Oakes.