Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records

Hypoglycemia is a common and serious complication affecting people with diabetes (1). It is an inappropriately low blood glucose (BG) that results in significant morbidity in people with type 1 diabetes and in many people with type 2 diabetes (2). A BG level of ≤3.9 mmol/L is defined as level 1 hypoglycemia. A BG level of 2.9 mmol/L and lower is defined as level 2 hypoglycemia requiring immediate action, as at that level, neurogenic and neuroglycopenic symptoms begin to occur (3). Hypoglycemia can lead to permanent neurological damage if not treated promptly and can ultimately be fatal (1).

Hypoglycemia is an important and common clinical problem under inpatient settings. Retrospective analysis of a U.S. electronic medical records database showed a 20% incidence of hypoglycemia and a 7% incidence of severe hypoglycemia (4). There was an associated 66% increase in adjusted inpatient mortality risk and greater than 50% increase in length of hospitalization stay. In a cross-sectional national audit of over 200 hospitals in the U.K., the 2017 National Diabetes Inpatient Audit showed that almost one in five people with diabetes experience hypoglycemia during their hospital stay. Although only 7% experience a severe (level 2) hypoglycemic episode, this rises to 26.9% of all patients with type 1 diabetes with 185 people over the course of 1 week requiring injectable rescue treatment for their hypoglycemia. Inpatient hypoglycemia has been implicated in the development of adverse clinical and economic outcomes, including increased mortality (5–7), adverse cardiovascular outcomes (8,9), and increased duration of hospital stay (6,10,11).

A recent review article has highlighted the urgent need for evidence-based methodologies to reduce inpatient hypoglycemia. Several strategies have already been developed to predict and prevent the occurrence of inpatient hypoglycemia (12). One approach to reducing inpatient hypoglycemia is to retrospectively analyze historical clinical data and develop a prediction tool to determine the individualized risk of hypoglycemia during an inpatient admission. With such a prediction tool, prevention measures can be tested in inpatients with high hypoglycemia risk. The possibility of developing such a prediction tool lies in the growing availability of rich clinical data sets stored in a hospital’s electronic patient records (EPR) system.

Previous studies have used clinical information from local healthcare systems to develop inpatient hypoglycemia risk prediction tools (13–15). In one study, the prediction model developed by the researchers was tested in a clinical trial. This demonstrated the feasibility of using such a model to reduce severe hypoglycemia (glucose <40 mg/dL or <2.2 mmol/L) in inpatients with diabetes (14).

However, previous studies have only applied multilinear or logistic regression models on these data sets resulting in only a modest predictive capability of the models. Over the last few years, a number of advanced machine learning techniques have been developed within the field of biomedical engineering (16). These can be used to create predictive models which can be tested and compared with traditional logistic regression models in order to determine the model with the best predictive performance. This is the first study to assess the performance of novel machine learning models in predicting the risk of inpatient hypoglycemia.

We compared the performance of 18 different machine learning algorithms in predicting the risk of hypoglycemia in inpatients with diabetes.

The study was approved by the Oxford University Hospitals National Health Service Foundation Trust Clinical Data Warehouse Program Board following completion of a Data Protection Impact Assessment. Data from Oxford University Hospitals National Health Service Foundation Trust was used which included the Cerner EPR system, the laboratory information management system and the point-of-care testing system. All the data used was collected for routine patient management with no additional data input required for the modeling. The data set contains hospital admission data from 1 September 2014 to 30 June 2018 for qualified patients with diabetes. The qualified patients are defined as meeting the following criteria: 1) being an inpatient as coded in the EPR; 2) having one diagnosis code among E10 (type 1 diabetes), E11 (non–type 1 diabetes), E13 (other specified diabetes), E14 (unspecified diabetes), or O24 (diabetes in pregnancy) as defined in the World Health Organization ICD-10 (17); and 3) having at least one BG test performed during the hospital admissions. Hospital admission data for qualified patients, including patient demographics, procedures undertaken, diagnosis, laboratory tests, medication administration details, and vital signs, were extracted from different source data systems and pooled into a final data table for use by the machine learning prediction models. A schematic representation of the data flow from the source data systems to the final data set used in the current study is depicted in Supplementary Fig. 1.

The outcome of interest in the current study is the risk of inpatient hypoglycemia during a hospital admission. We prepared two binary outcome variables Hypo^<4.0 and Hypo^<3.0 for each hospital admission representing two different severities of the potential hypoglycemic episodes since the degree of hypoglycemia may be influenced by different clinical predictors. We put value 1 to Hypo^<4.0 for any level 1 hypoglycemic episode (any BG measurement <4 mmol/L) and value 1 to Hypo^<3.0 for any level 2 hypoglycemic episode (any BG measurement <3 mmol/L) detected during the hospital admission, and value 0 to the two variables if no hypoglycemic episode detected (all BG measurements >4 mmol/L).

We preprocessed the integrated data set from the EPR and prepared 42 candidate predictors of interest based on clinical knowledge and previous studies. The predictors cover patient demographics (age, sex, and so on), procedures (value 1 for at least one procedure undertaken), laboratory test results (sodium, potassium levels, and so on), medication administration details (names and doses of medication delivered including different types of insulin), and vital signs (temperature, heart rate, and so on). Additional variables were added to the data set to improve the performance of the prediction algorithm. These included an episode of hypoglycemia in a previous admission within 6 months. Table 1 shows the full list of predictors and how they were represented in the source data systems and in the prediction models with units of the predictors provided.

We evaluated the prediction performance of 18 different machine learning models on the data set. The models were used to predict the risk of hypoglycemia (either BG <4.0 mmol/L [Hypo^<4.0] or <3.0 mmol/L [Hypo^<3.0]). A total of 42 different variables were used as inputs into the prediction model. The models cover a wide range of commonly used classification algorithms including logistic regression, random forests, and artificial neural networks that have been previously demonstrated to be robust and applicable to big data sets.

For internal model validation, we used a 10-fold cross-validation. We randomly selected nine-tenths of the data set to be the training data set (developing the model) and the remaining one-tenth to be used to validate the model.

The outcome variables indicate whether or not a hypoglycemic episode occurred during an admission. The model was constructed to predict the probability of at least one hypoglycemic episode occurring. We measured the area under the receiver operating characteristic curve (AUROC), which shows the probability that the model correctly ranks the risk of hypoglycemia) higher no hypoglycemia. The AUROC is not sufficient on its own to use as a hypoglycemia prognostic model as it does not take into account the prevalence of hypoglycemia in the population. It assumes that positive and negative predictions are equally important. A detailed technical analysis of the shortcomings of the AUROC was recently conducted by Saito et al. (18). To ensure a comprehensive assessment of predictive performance, we used additional metrics. We used the terminology TP to represent the number of true positive predictions; FP to represent the number of false positives; TN to represent the number of true negatives; and FN to represent the number of false negatives. We defined precision (positive predictive value) as the ratio TP/(TP + FP). This is a measure of the ability of the model to correctly predict a patient as having hypoglycemia. We defined recall (sensitivity) as the ratio TP/(TP + FN). This is a measure of the ability of the model to label as hypoglycemic all of patients who did indeed develop hypoglycemia. The precision and the recall were calculated for each machine learning model (Table 3).

Following the development of the initial model (inpatient hypo [IH]), two further iterations of the data set were carried out. In the first change to the data set, the dose of insulin was added and the variable of intravenous insulin was indicated separately (IH+). In the second change, any previous admission in the last months containing a low glucose (<3 and <4 mmol/L) were added (previous hypo [PH]). The 18 different machine learning models were then rerun on each of these new data sets, and the model outputs were compared with the original data set. The last column in Table 1 shows which variable is used in each data set.

We sought to understand how the different variables contributed to the predictions by the XGBoost model (the best predictive model among the 18 tested models). We evaluated the predictive power of each individual variable by providing XGBoost with one variable at a time and assessing the diagnostic accuracy of the model that it constructed using only that variable. We evaluated the AUROC (using 10-fold cross-validation) to get a full picture of each variable’s predictive power.

All statistical analyses were performed using Python 3.6 and R version 3.3. All algorithms were implemented with the machine learning library “sklearn” (19) that contains all the algorithms and data science utilities used in this report. Internal validation was obtained via 10-fold stratified cross-validation. Performance comparisons were made with t tests from which we specified P values <0.001 to be considered statistically significant.

We analyzed data obtained from 17,658 inpatients with diabetes (9,277 males, mean [SD], age 66 [18] years) who underwent 32,758 hospital admissions between July 2014 and August 2018. We identified all the biochemical (level 1) and clinically significant (level 2) hypoglycemic episodes during these admissions. The incidence of biochemical hypoglycemia during a hospital admission was 21.5% and that of clinically significant hypoglycemia was 9.6%. This is in keeping with data from the National Diabetes Inpatient Audit (20).

A selection of the baseline characteristics of the inpatient cohort and the glycemic outcomes are reported in Table 2.

The performance metrics of the machine learning models tested on the PH data set are presented in Table 3. The AUROC varied between 0.62 and 0.96 for different machine learning models. The estimation performance was better when predicting the risk of Hypo^<3.0 compared with predicting that of Hypo^<4.0. The AUROC for the logistic regression model was acceptable with 0.73 and 0.75 for estimation of the risk of Hypo^<4.0 and Hypo^<3.0, respectively. However, the best performing model for predicting the risk of Hypo^<4.0 and Hypo^<3.0 was the XGBoost model, which had the highest AUROC (0.96 for both), the highest precision value (0.88), as well as a high recall value (0.70) among all the models. Figure 1 shows the ROC curves contrasting the logistic regression, gradient boosting, decision tree, and XGBoost models. Supplementary Table 4 shows the normalized confusion matrix for these four models with the true positive and true negative rates on the upper left and lower right in the matrices, respectively. The XGBoost model was again the best performing model with a true positive rate of 0.98 and a true negative rate of 0.71 (Supplementary Table 4).

Estimated regression coefficients with SEs and P values from the logistic regression model with the PH data set are presented in Supplementary Table 3. The variables that are significant predictors of hypoglycemia are shown in Table 4. Similar predictors for Hypo^<4.0 and Hypo^<3.0 were found. Significant predictors for both Hypo^<4.0 and Hypo^<3.0 included weight, type of diabetes, oxygen saturation, albumin level, sulfonylurea use, metformin use, intravenous insulin titration, long-acting human insulin use, procedures undertaken, and previous hypoglycemic episode.

Two iterations of the original data set were carried out during the development of the prediction model. Supplementary Table 1 shows that the addition of a variable to discriminate patients on intravenous insulin compared with those on subcutaneous insulin and a variable for the dose of subcutaneous insulin (IH + data set) increased the best performing model (XGBoost) AUROC by 3 percentage points for Hypo^<4.0. When the PH data set was used, all models showed higher AUROC values while the XGBoost and gradient boosting models stood out with a significant increase of AUROC by 15 percentage points.

Supplementary Table 2 demonstrates the relative importance of the variables with the top three most important variables being previous hypoglycemic episodes, albumin levels, and type 2 diabetes. A number of novel predictive variables were identified from the machine learning method. These include several vital signs and medications that have logical clinical rationale underlying their importance to hypoglycemia. Further studies will be required to confirm their importance in the development of hypoglycemia.

To our knowledge, this is the first study comparing the performance of advanced machine learning models in predicting the risk of inpatient hypoglycemia. With the rich inpatient data set collected from the EPR system in a large university hospital, the 18 machine learning models showed high predicting power with an average AUROC at 0.85 for the detection of clinically significant hypoglycemia. The model with the highest AUROC (0.96) was the XGBoost model, which outperformed the logistic regression model. This model performed equally well in predicting both level 1 hypoglycemia (BG <4 mmol/L) and level 2 hypoglycemia (BG <3 mmol/L).

Among the 18 evaluated machine learning models, most machine learning prognostic models performed markedly better than the predictions of the linear regression model. XGBoost is a highly flexible nonparametric model that integrates a large number of other machine learning models (decision trees). It was consistently the best performer with the highest AUROC, the highest precision, and good recall. It is important to note that XGBoost performed significantly better than the linear regression model, which has been used to predict inpatient hypoglycemia. There was an improvement of over 20 percentage points in terms of AUROC for prediction of both biochemical and clinically significant hypoglycemia. The high predictive capability of the XGBoost model also came with a high precision and recall showing low levels of overestimation of risk and low levels of missed events.

With the stepwise iteration of the data set, the predictive ability of the XGBoost model and some other models improved significantly. This was in contrast to the logistic regression model, which showed no significant improvement. This emphasizes the importance of developing a clinically relevant and comprehensive data set on which to base any machine learning in order to optimize the capability of the learning algorithm.

Our data set covers a wide range of potential predictors of inpatient hypoglycemia. The logistic regression model detected a number of significant predictors for clinically significant hypoglycemia including weight, type of diabetes, diastolic blood pressure, oxygen saturation, temperature, albumin levels, sulfonylurea use, metformin use, intravenous insulin use, high dose of long-acting human insulin, and people undertaking procedures (see Supplementary Table 3). Previous studies also found similar predictors, such as the albumin levels and glucose-lowering drugs (13,15,21). In our data set, undertaking any kind of procedure was also found to be a significant predictor. This is clinically understandable as procedures may disrupt the daily hospital routine of food intake and drug administration and, thereby, cause increased variability in glycemic levels.

Our machine learning models outperform other inpatient hypoglycemia prediction models that have been published using logistic or multivariate regression techniques. These have shown a discrimination of between 0.70 and 0.80 (15,21). Our model development also includes a variable for previous hypoglycemia, which has not previously been included in other prediction models.

Machine learning models have been widely used within hospital information systems to predict the risk of emergency admission, sepsis in the intensive care unit, and identifying type 2 diabetes using electronic health records (22–24). The performance of the current study also compares favorably to these other predictive models that have an AUROC of between 0.75 and 0.85.

There are several key strengths of the current study. First, we evaluated a wide range of machine learning models and compared their predicting ability against the most commonly used statistical model, which we used as a benchmark model. Second, we used an iterative approach to develop the model with additional variables that revealed how significant improvements to the model could be achieved. Third, this was the largest data set used to predict inpatient hypoglycemia containing the data for 32,758 hospital admissions. It also integrated clinical information that previous studies have not considered before, such as the medication dosage information and hypoglycemia in previous admissions.

However, as with all modeling efforts, there are also limitations. One limitation with the data set is the unavailability of carbohydrate intake/meal content information from the EPR during the hospital admissions. Carbohydrate intake has a direct impact on the postprandial glucose excursions and, consequently, the prandial insulin doses titrated to individuals and could be an important predictor for hypoglycemic events. This information is unavailable because it is not routinely recorded electronically in hospitals. Second, the current electronic health record does not record the level of hypoglycemia unawareness, prior continuous glucose monitoring (CGM) data, or prior self-monitoring of BG. These data are likely to be an important factor in developing hypoglycemia while in hospital, although for acutely unwell patients, these data may not be directly applicable. Third, our data set is derived from a single organization, and the generalizability of our best performing machine learning model needs to be tested in other data sets and evaluated in other centers. Finally, although we have developed a highly predictive model for inpatient hypoglycemia, the feasibility of using this model needs to be tested within a live EPR to confirm the ability of the model to receive data in real time and ensure that the model performs as strongly as the current data suggests.

One of the advantages of the current prediction model is that it uses variables that are readily accessible within the EPR. As a result, the model can be integrated into a decision support system under the EPR framework. In practice, the decision support system would access the clinical information of a new inpatient and feed the required information to the prediction model, which would then calculate the risk of the patient experiencing either biochemical or clinically significant hypoglycemia during his or her hospital admission. This would enable the decision support system to suggest appropriate treatment options based on individual risk levels, thereby reducing hypoglycemia and its consequent associated morbidity and potentially reduce the economic burden of prolonged hospital stay due to hypoglycemia. In a previous single-center study, a linear regression prediction model with a sensitivity of 50% and specificity of 71% was used to detect patients at risk for hypoglycemia. Clinician education resulted in medication change in 40% of patients and a reduction of 68% in the rate of severe hypoglycemia in alerted high-risk patients versus nonalerted high-risk patients (14). The benefits of a significantly more powerful prediction model based on machine learning need to be evaluated in a large multicenter randomized controlled trial.

Another potential application of the prediction model is the selection of high-risk patients who may benefit from an advanced treatment option, such as CGM or closed-loop insulin delivery. The use of CGM in the inpatient setting was considered at a recent symposium where the trials in both the intensive care unit and non–intensive care unit settings were reviewed (25). While there was some evidence that CGM may reduce the rates of severe hypoglycemia, it was recognized that there were limited data on clinical outcomes and that such technology may be most suitable for “populations.. at high risk for glucose variability and hypoglycemia.” Closed-loop insulin delivery, or artificial pancreas, is a novel treatment option for people with diabetes who require exogenous insulin administration (26). The system titrates insulin based on real-time glucose monitoring and a titration algorithm. Previous clinical studies have shown promising glycemic results of the closed-loop systems under inpatient settings (27,28). However, the system is costly and cannot be applied in every inpatient with diabetes. Prediction models, such as the one described in this study, could be used as a preselection tool to determine which patients would benefit the most from CGM or automated insulin delivery.

In conclusion, this study demonstrates for the first time, the utility of advanced machine learning models in predicting the risk of hypoglycemia for inpatients with diabetes. We have shown that these models are significantly better in predicting inpatient hypoglycemia than the traditional logistic regression model. However further trials are needed to determine if this prediction model provides a significant clinical advantage over traditional logistic regression analysis or more simple risk factor prediction models, e.g., insulin use alone. Such machine learning models need to be evaluated within a real-time clinical setting to demonstrate their ability to predict hypoglycemia following admission. The use of new technological methods, such as machine learning and artificial intelligence, are not a substitute for clinicians, but they should be used to enhance clinical judgement and support everyday decisions. Multicenter clinical trials are now needed to evaluate their utility within a clinical decision support system and reduce the burden of hypoglycemia in hospital.

Funding. Y.R. received salary funding from a Novo Nordisk Postdoctoral Fellowship run in partnership with the University of Oxford. G.D.T., A.L., J.D., and R.R. are partly funded by the National Institute for Health Research Oxford Biomedical Research Centre.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. Y.R. and A.B. carried out the data analysis. Y.R. and R.R. designed the study analysis and drafted the manuscript. All authors contributed to the interpretation of the results and critical review of the manuscript. Y.R. and R.R. are the guarantors of this work and, as such, had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.