A Prediction Model for Adverse Outcome in Hospitalized Patients With Diabetes

The setting is University Hospital Birmingham, a tertiary hospital providing secondary care services to an ethnically diverse population in the West Midlands, U.K. Data were derived from the Patient Administration System (PAS) and the Patient Information and Communication System (PICS) and included inpatient admissions from 2007 to 2010. The PAS database records demographic details, admission and discharge time, discharge diagnostic codes, and death. The PICS database contains patients’ clinical pathology test results, electronic observations, and prescriptions. The PAS database was linked to the PICS database using individual patient identification numbers and admission time, and patients with both records were included in the analysis (18). Patients were identified as having diabetes using both discharge diagnostic codes and prescription data as published previously (18).

The outcome is a composite of either excessive length of stay or death.

Initially, all admissions (both with and without diabetes) were categorized into 260 groups of clinical conditions based on primary diagnosis from the discharge diagnostic code. The 260 groups are defined as per the Agency for Healthcare Research and Quality clinical classification system and were recently adopted by the National Health Service Information Centre (19). The median length of stay was derived for each clinical condition for nondiabetic patients. The excess length of stay for each admission with diabetes was defined as the difference between the actual length of stay and the median length of stay for nondiabetic patients with the same group of clinical conditions. An excessive length of stay is defined as an excess length of stay >75th centile of all diabetic admissions. This cutoff was selected because 25% of admissions with diabetes accounted for 85% of excess length of stay in patients with diabetes. The cutoff point also corresponds to 6 days more than what would be expected for any given group of clinical conditions in a nondiabetic patient.

Our methodology avoids the need to know the condition with which the patient is admitted; this is important because the diagnosis (and hence group of clinical conditions) may not be clear at the time of admission.

Three models are presented: a pragmatic model, a test model, and an ideal model. The pragmatic model, is intended to be used to predict adverse outcomes early in admission. It uses clinical pathological test results instead of a measure of comorbidity because diagnoses may not be available at or around the time of admission. The test model replaces the clinical pathological test results from the pragmatic model with a measure of comorbidity (modified Charlson comorbidity score) to determine whether these clinical pathological test results are a good alternative for measuring case-mix. The ideal model includes all variables available at discharge and so includes clinical pathological test results, modified Charlson comorbidity score, and a measure of deprivation.

Prognostic factors included in the pragmatic model are those that can be reasonably expected to be available within the first 72 h of admission. Data that are readily available include age, sex, ethnicity, and admission type (emergency or elective). The presence of foot disease is also included in the pragmatic model because recommendations suggest that it should be recorded in the first 24 h of admission (8). We assume that patients prescribed insulin during their stay had this initiated within 72 h of admission. Similarly admissions to an intensive care unit (ICU) setting are mainly within the first 72 h (75% at 72 h, 68% at 48 h, and 52% at 24 h in our database). Therefore, both of these variables are included in the pragmatic model. Clinical pathological test results that predict adverse outcomes in hospital settings are also included. These include albumin (20,21), estimated glomerular filtration rate (GFR) (22), serum sodium (13,14), serum potassium (23), C-reactive protein (CRP) (15), and hemoglobin (16). White cell counts are also a valid predictor (24), although we decided to use neutrophil counts instead, as this is a better predictor of infection and/or sepsis (25).

Although blood glucose concentrations predict poor clinical outcome (17,26–29), we were not able to include admission glucose concentrations in the pragmatic model as 74% of glucose measurements were missing. Information on comorbidities may not be available to extract from hospital information systems at the time of admission, and therefore, comorbidities are not included in the pragmatic model but are included in both test and ideal models. Deprivation levels are difficult to compute at or around the time of admission, and also definitions differ across countries and were therefore excluded from the pragmatic model but included as part of the ideal model.

Foot diseases were defined based on National Institute for Health and Care Excellence (NICE) guidelines using ICD-10 codes and OPCS4 codes (a U.K. classification system for interventions and procedures) that were available as a discharge diagnostic code in the database (8). These included foot ulcers, peripheral vascular disease, peripheral neuropathy, arthropathy, and operative procedures such as wound debridement and amputation. Codes for foot ulcers were similar to a national report recently published in the U.K. (30), except we excluded the codes indicating decubitus ulcers. These codes were excluded as pressure ulcers could be either in the sacrum, buttock, or heel (31,32), are less specific, and are less likely to occur early in an admission. Furthermore, patients with decubitus ulcers are known to have poor clinical outcomes (33), and, as a result, estimated effect size for the foot disease will be biased if these codes were included. Modified Charlson comorbidity score (27) was calculated by excluding the scores linked to diabetes from the Charlson comorbidity score (11). Deprivation quintiles were defined using disaggregated income component of the Indices of Multiple Deprivation score (34).

The first value of the clinical pathology tests for each admission was included in the analysis. Most were available (94%) within the first 72 h of admission. Cutoff points to categorize the clinical pathological results were based on 1) normal ranges, 2) definitions of severity for a given marker [i.e., GFR reflecting stages of renal disease as per guidelines (35)], 3) adequate number of groups to illustrate any dose-response relationship, and 4) sufficient numbers in each group to observe any meaningful results.

The demographic characteristics, clinical pathology test results, and morbidity characteristics of the patients with and without the adverse outcome are summarized using means (SD) or medians (interquartile range) as appropriate for continuous data and using proportions for categorical data.

Odds ratios (ORs) (95% CIs) were estimated using logistic regression models. To account for the clustering effect created by the same patient being admitted more than once, we used population-averaged methods (36).

In the multivariable model, prognostic factors were selected based on their clinical significance as identified from the literature and were therefore preserved in the model irrespective of their statistical significance (37,38). This meant we did not carry out any stepwise procedures. Neither did we use any interaction terms, as these add complexity and are difficult to use in clinical practice.

Data were complete for all variables other than for clinical pathology test results. For these variables, to allow for the missing data, multiple imputations (10 imputations) were carried out using the Multiple Imputations by Chained Equations with predicted mean matching. The chained equations allow imputation of both categorical and continuous variables, while predicted mean matching restrict imputations to the appropriate range (by matching the predicted value to the closest value in the dataset).

Model performance is assessed by its ability to distinguish between those with and without the outcome of interest (discrimination) and by looking at the agreement between the frequency/proportion of observed and predicted outcomes (calibration). In addition, the sensitivity, specificity, positive predictive values, negative predictive values, and likelihood ratios were determined. To assess discrimination, a receiver operating characteristic curve was constructed and area under the curve (AUC) (Harrell’s C-statistics) was calculated. C-statistics were compared between the three models (pragmatic model, test model, and ideal model) using nonparametric tests (39). Calibration was assessed by plotting predicted probabilities of outcome, by decile groups, against observed probabilities of outcome (in same decile group), and by overlaying a smoothed calibration curves (using lowess algorithm) to judge against a linear line (37).

In a sensitivity analysis, models were internally validated using bootstrapping. This process repeatedly samples, with replacement, samples of datasets from the original dataset and reestimates model parameters in each resampled dataset. Parameter estimates are then averaged over all of the resampled datasets. We found that results from the internally validated model and the primary model differed very little, and so, we report only the results from the primary model. This process will ensure that we identify any evidence of overfitting to the original dataset (that is producing estimates that are a good fit to the developmental data but perform poorly in prediction) and accommodate this by reporting shrunken regression coefficients (38). To implement this for each of the 10 imputed datasets, we applied a bootstrap procedure (2,000 resamples) to obtain 10 sets of shrunken regression coefficients (with SEs) and C-statistics (with SEs), which we then combined using Rubin’s rule (40).

Coefficients, model performance, and sensitivity analysis are reported for the pragmatic model only. For the other two models (test model and ideal model), the AUC is reported. All data analyses were carried out using Stata 12 software (Stata Statistical Software, release 12; StataCorp LP, College Station, TX).

Out of the 171,067 admissions (2007–2010) included in the analysis, 25,118 (14.7%) had diabetes. Out of these 25,118 admissions, 6,281 (25%) were categorized as having an excessive length of stay and 1,286 (5.1%) died. Excluding overlaps, this meant 6,928 (28%) had an adverse outcome. Among these 25,118 admissions, 10,596 (42%) admissions had an expected length of stay less than or equal to the median length of stay for the same presenting clinical condition of nondiabetic patients. The remaining patients (N = 14,522; 58%) contributed to 146,680 excess days with a median of 4.7 days and a mean of 10.1 days. The 6,281 admissions identified as contributing markedly to excess length of stay accounted for 85% (124,803 days) of the excess days.

Admission characteristics and clinical pathology test results of those with and without adverse events are given in Tables 1 and 2. Coefficients and the ORs of the prognostic factors used in the pragmatic model are given in Table 3. In the pragmatic model, there was an incremental rise in the odds of adverse outcome with increasing age. The oldest age group (≥85 years) had an OR of 5.64 (95% CI 4.66–6.81) in comparison with the youngest (16–44 years). Females had slightly higher odds of having an adverse event (1.08 [1.00–1.16]), while those of Asian ethnic minority (largely South Asian ethnicity in this region of the U.K.) had a significant lower chance of having an adverse event (0.86 [0.78–0.94]). Those admitted as an emergency (2.94 [2.69–3.21]), on insulin therapy (1.89 [1.76–2.03]), or with foot disease (2.46 [2.16–2.80]) were at higher odds of having an adverse event. Receiving treatment in an ICU setting had the highest OR (10.79 [9.52–12.22]).

Severe hyponatremia (<125 mmol/L) had higher odds (1.71 [1.26–2.32]) of having an adverse outcome than mild to moderate (125–134 mmol/L) hyponatremia (1.17 [1.06–1.28]). Similarly, the greater the severity of hypernatremia, the larger the effect size was (145–154 mmol/L, 1.31 [1.12–1.54]; and >155 mmol/L, 4.05 [1.84–8.91]). Such a dose-response relationship favoring an adverse outcome was also noted with lowering hemoglobin concentrations (anemia) and hypoalbuminemia and rising CRP concentrations and neutrophil count. Hypokalemia (1.79 [1.34–2.39]) but not hyperkalemia (1.00 [0.80–1.26]) was associated with the adverse outcome. A GFR <30 mL/min/1.73 m² had an OR of 1.31 (95% CI 1.15–1.48) in comparison with normal GFR (≥90 mL/min/1.73 m²).

The pragmatic model had an AUC of 0.802 (95% CI 0.795–0.808), performing significantly (P < 0.001) better than the test model (AUC 0.784 [95% CI 0.777–0.790]), suggesting that the clinical pathology results replaced comorbidities as a measure of case-mix well. The ideal model performed better than the pragmatic model (0.810 [0.804–0.816]; P < 0.001), but the difference between the pragmatic model and ideal model was minimal (AUC 0.802 vs. 0.810) (Fig. 1).

The sensitivity and specificity total was maximal at a cutoff of >25% predicted chance of an adverse event. In this study, the sensitivity was 76%, specificity was 70%, and the positive predictive value was 49% (Supplementary Fig. 1). However, in reality, patients in ICU will not be part of the active case-finding approach, and if one were to see only those in the noncritical care setting, the approach will have a sensitivity of 69%, specificity of 72%, and a positive predictive value of 43%. In contrast, the so-called false-positive patients in noncritical care whom will be identified in an active case-finding approach have characteristics such as high comorbidity index (41% had modified Charlson comorbidity score of ≥2), insulin use (66%), foot disease (10%), and age >75 years (51%). Therefore, false positives are not necessarily admissions that will not benefit from enhanced models of care.

The lowess calibration plot was in close proximity to a line drawn at 45°, suggesting the calibration of the model was good; i.e., the predicted probabilities were similar to that of the observed probabilities (Supplementary Fig. 2).

The sensitivity analysis revealed that there was little evidence of overfitting. This was demonstrated both by minimum differences between regression coefficients (Table 3) and also by little difference between C-statistics. For example, the internal validated bootstrapped sample had an AUC of 0.798 (95% CI 0.792–0.805), only a marginal difference to that observed in the pragmatic model (0.802 [0.795–0.808]).

We show that predictor variables, routinely available at the time of admission, facilitate the identification of diabetic patients at risk for an excessive length of stay or death. At a cutoff value of 25% probability of having an adverse outcome, the sensitivity was 76%, specificity was 70%, and the positive predictive value was 49%. The pragmatic model performed well with an AUC of 0.802 with only a small reduction being noted in the internal validation (AUC 0.798 [95% CI 0.792–0.804]).

The association of the clinical pathology test results with length of stay or death in our analysis of diabetic patients corresponds with those previously identified for unselected patients. These include hypernatremia (13,14), anemia (16), hypokalemia (23), and raised CRP concentrations (15). Our data validates these findings in people admitted to hospital with diabetes. Furthermore, some of these criteria may be more relevant in the context of diabetes: sodium and potassium imbalance can challenge the management of diabetic ketosis; anemia and reduced renal function may indicate advanced diabetic nephropathy; and an elevated CRP and neutrophilia may indicate compromised sepsis/wound healing in diabetes.

We believe our model will have important clinical utility. Timely identification of high-risk patients provides the opportunity for early intervention and improvement in clinical outcome. In our hospital, we plan to incorporate the model as part of decision support within the electronic medical records. On identification of high-risk patients, an automated consultation request would be generated to the inpatient diabetes team. They can then be reviewed early in their admission. We believe this will be beneficial, as there is clear evidence that specialist nurse review can reduce length of stay (6,7). Other potential uses include redesigning care models and pathways for patients with diabetes admitted to hospital. This is particularly important with planning for seasonal fluctuations in hospital admission rates.

There are limitations to our approach. First, we have assumed diabetes was present at the time of admission and that all admissions with diabetes can be identified through information systems. The latter may not be possible in all hospitals with electronic health information systems. However, we have previously shown this is achievable using either mandatory entry of the diabetes status on admission or in a comparable way using electronic prescription data alone (18). Second, blood results may not be available at the time of diagnosis, and these therefore will be categorized as normal values when using the prediction model. Although this may compromise the validity of the model, additional sensitivity analysis undertaken by replacing the missing categories with normal categories and applying the coefficients obtained from the multiple imputation model to the dataset suggested better performance than shown (AUC 0.816, sensitivity 73%, specificity 75%, and positive predictive value 53%) (Supplementary Fig. 3). We were not able to differentiate between type 1 and type 2 diabetes, as the recorded ICD-10 codes are not reliable discriminators. The strength of our study is that the model is developed in a hospital with a diverse population from a large dataset with effect sizes that had narrow CIs and rigorous methodological quality. Furthermore, our definition of excessive length of stay is novel and mitigates the need to know the presenting condition. Similarly, our model does not rely on knowing the comorbidities, replacing this instead with routinely performed blood tests.

We conclude that our active case finding model is a useful tool to identify patients with diabetes who may be at risk for poor outcomes such as increased length of stay and death. Further studies should aim to 1) externally validate the model, 2) assess the practicality of using the model, and 3) demonstrate if the active case-finding approach either on its own or in combination with additional clinical indicators of poor outcomes (such as hypoglycemia, hyperglycemia, and insulin infusions identified through electronic records), followed by appropriate care (e.g., review by the diabetes specialist nurse) will positively impact on reducing adverse outcomes for patients with diabetes.

This study was funded by the National Institute for Health Research through the Collaborations for Leadership in Applied Health Research and Care for Birmingham and Black Country programme.

No potential conflicts of interest relevant to this article were reported.

K.N. was responsible for conception and design, analysis and interpretation of data, drafting the article, and revising it critically for important intellectual content. K.H. was responsible for analysis and interpretation of data and revising the article critically for important intellectual content. P.N. and T.M. were responsible for interpretation of data and revising the article critically for important intellectual content. J.J.C. was responsible for conception and design, interpretation of data, and revising the article critically for important intellectual content. All authors gave approval for the final version to be submitted. K.N. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.