Natural Language Processing Improves Detection of Nonsevere Hypoglycemia in Medical Records Versus Coding Alone in Patients With Type 2 Diabetes but Does Not Improve Prediction of Severe Hypoglycemia Events: An Analysis Using the Electronic Medical Record in a Large Health System

The risk of hypoglycemia (glucose <70 mg/dL) with diabetes treatment is well recognized, and its prevention is considered a “critical component of diabetes management” (1). Episodes of severe hypoglycemia (SH), defined in this study as requiring hospitalization or emergency department visit in patients with type 2 diabetes, can be identified through structured electronic medical record (EMR) or administrative data (2,3). However, nonsevere hypoglycemia (NSH), not requiring assistance for recovery (1), is often reported by patients during outpatient visits and captured only if an EMR hypoglycemia diagnosis code is entered. Natural language processing (NLP) combines computer science and linguistics methods (4) and can be used to extract nonstructured EMR text data. The improvement of EMR hypoglycemia detection using NLP had been previously reported (5). We aimed to describe capture of NSH in patients with type 2 diabetes using both diagnosis codes and NLP and to create a predictive model for SH events.

We used a modified version of the algorithm used by Kho et al. (6) to identify patients in the Cleveland Clinic Health System with type 2 diabetes during 2005 to 2017. Hypoglycemia events were identified using ICD-9/ICD-10 codes (Supplementary Appendix 1); events not associated with hospitalization/emergency department visit were categorized as NSH. A randomly selected subset of patient records stratified by years was reviewed to confirm code identification of NSH. The study protocol was approved by the Insitutional Review Board at Cleveland Clinic (Cleveland, OH).

Using clinical progress notes, an NLP algorithm was developed. The cTAKES program (7) was used to break down sentences into phrases to identify hypoglycemia-related Unified Medical Language System concepts. Regular expressions were written to classify polarity (event or no event) of phrases by pattern matching. Using word embeddings, the remaining phrases were classified with additional pattern matching (see Supplementary Appendix 2 for detailed descriptions). For each year, 100 events were randomly selected, with 2005 to 2006 combined given the lower number of notes. The phrases were reviewed by practicing clinicians and confirmed through record review of a subset of patients as representing true events. The weighted proportion of actual events in a set of 1,200 notes was used to define the positive predictive value.

We then limited our data set to include patients with at least one primary care or endocrinology department visit each year and created a Cox proportional hazards prediction model for SH (5 years’ duration) using the following variables based upon our previous work (2): sex, race, median income (8), history of comorbidities (cardiovascular disease, congestive heart failure [CHF], depression, other psychiatric disorders, dementia, cognitive impairment, chronic kidney disease [CKD], and alcohol or substance abuse), and time-dependent varables including age, insurance type (Medicare, Medicaid, commercial, or other), glycosylated hemoglobin (HbA_1c), BMI, and diabetes medications (insulin, sulfonylurea, glucagon-like peptide 1 receptor agonists [GLP-1RAs], sodium–glucose cotransporter 2 inhibitors, dipeptidyl peptidase 4 inhibitors, metformin, and α-glucosidase inhibitors). We had previously identified NSH by ICD codes as a variable associated with SH but for this study added two variables: ever-history of NSH and history of NSH within the past 3 months detected by codes and NLP. We also included interaction variables for HbA_1c, with insulin, and with sulfonylureas.

The number of patients with type 2 diabetes in this data set increased from 12,706 in 2005 to 176,001 in 2017 (210,191 unique patients). A total of 1,177,590 progress notes were processed. Upon clinician chart review, 1,111 of 1,200 randomly selected events classified as NSH by NLP were confirmed (93% positive predictive value). From 2005 to 2017, 10,205 NSH events were captured by codes and 14,763 events by NLP, with overlap of only 5 events. Among 204,517 patients with no codes for NSH, evidence of NSH was found in 7,035 (3.4%) using NLP.

The chart review confirmed NSH episodes were often documented under general diabetes diagnosis categories. The incidence proportion of patients with SH by ICD codes increased from 0.3 to 1.7% and those with NSH increased from 0.4 to 1.3% from 2005 to 2017. When NLP was added, the incidence proportion of patients with NSH increased from 0.8% (2005) to 2.6% (2017).

There were 47,280 patients included to create the prediction model for SH (Supplementary Appendix 3). The models using NSH codes alone and codes plus NLP (Table 1) had C-statistics of 0.812 and 0.806, respectively. The codes plus NLP model showed increased risk for SH with NSH (ever-history of NSH: hazard ratio [HR] 4.44, P < 0.001; and NSH in past 3 months: HR 1.65, P < 0.001), black race (HR 1.81; P < 0.001), Medicaid insurance (HR 1.35; P = 0.008), history of cardiovascular disease (HR 1.58; P < 0.001), CHF (HR 2.35; P < 0.001), depression (HR 1.28; P = 0.03), psychiatric disorders other than depression (HR 1.55; P < 0.001), alcohol or substance abuse (HR 1.55; P = 0.04), and CKD (HR 1.86; P < 0.001). The effect of insulin use on SH was greater at lower HbA_1c (HR of 3.32 when HbA_1c is 6% [42 mmol/mol]; HR of 2.04 when HbA_1c is 9% [75 mmol/mol]). There was decreased risk of SH with higher BMI (HR 0.69 when BMI was 30 kg/m²; HR 0.63 when BMI was 35 kg/m²) and decreased risk with metformin (HR 0.53; P < 0.001) and GLP-1RA (HR 0.36; P < 0.001). Sulfonylurea use had a mixed effect on the risk of SH (HR of 1.61 when HbA_1c was 6% [42 mmol/mol]; HR of 0.69 when HbA_1c was 9% [75 mmol/mol]). The effect of HbA_1c varied at the extremes. With a reference HbA_1c of 6% (42 mmol/mol), the HR for SH was 1.59 at an HbA_1c of 5% (31 mmol/mol), HR was 0.73 at an HbA_1c of 7% (53 mmol/mol), and HR was 1.39 when HbA_1c was 9% (75 mmol/mol). Of note, NSH within the past 3 months was only a significant predictor in the the model including NLP. Receiver operating characteristic curves comparing code-detected NSH, NLP-detected NSH, and code plus NLP–detected NSH models were similar (Supplementary Appendix 4), while Kaplan-Meier curves for probability of being free from SH showed lower probability for code-detected NLP (Supplementary Appendix 5).

Building upon previous work identifying SH in our health system, we found that applying NLP to progress notes improved NSH detection and that NSH is a significant predictor for SH. However, SH prediction using NSH diagnosis codes did not improve with adding NLP, possibly related to the strength of the other variables in contributing to the model and possible greater severity of coded NSH as the Kaplan-Meier curves suggest. There was little overlap between NSH events by codes plus NLP, suggesting that while NSH may be reported by patients, providers may not enter a hypoglycemia diagnosis code, highlighting the benefit of NLP.

While it is known that NSH may have significant adverse consequences (9), treatment deintensification is uncommon (10,11). Identifying patients with increased risk for hypoglycemia (12,13) and alerting providers may affect provider behavior to adjust treatment for at-risk patients (13). Our prediction model identifies demographic characteristics and comorbidities predicting increased SH risk. Modifiable predictors include history of NSH and sulfonylurea and insulin use predicting increased risk of SH at lower HbA_1c levels and metformin and GLP-1RAs predicting lower risk. The HbA_1c effect is of higher predicted risk overall at very low HbA_1c levels at which treatment deintensification may be relevant, but also at very high HbA_1c, at which fluctuating blood sugars may increase SH risk and limit ability to attain a lower HbA_1c.

While our data set is robust, we recognize these limitations: 1) the EMR from one health system does not capture NSH or SH events outside of the system, and 2) lack of diabetes duration information in EMR for prediction. Future work will focus on notifying providers at the point of care of captured NSH events to consider deintensification, medication adjustments to lower SH risk, or meal and activity timing, especially in patients with multiple comorbidities, to reduce risk of future SH.

Duality of Interest. This study was funded by Novo Nordisk, Inc. A.D.M.-H. has received research support from Merck, Novo Nordisk, Inc., Boehringer Ingelheim, and the Agency for Healthcare Research and Quality (K08-HS-024128) within the past 12 months. A.M. reports receiving research funding from Merck, Boehringer Ingelheim, Novartis, and Novo Nordisk, Inc., within the past 12 months. A.Z., X.J., J.M.B., and M.W.K. report receiving research funding from Merck and Novo Nordisk, Inc., within the past 12 months. T.D.H., W.W., M.M., and R.G. report being employees of Novo Nordisk, Inc., and holding company stock. P.P. and S.X.K. report being employees of Novo Nordisk, Inc., and holding company stock at the time of the study. K.M.P. reports receiving research funding from Novo Nordisk, Inc., and Merck; receiving consulting fees from Novo Nordisk, Inc., Sanofi, Eli Lilly and Company, and Merck; and participating in the speakers bureaus of Novo Nordisk, Inc., Merck, and AstraZeneca within the past 12 months. R.S.Z. reports receiving research funding from Novo Nordisk, Inc., and Merck and participating in the speakers bureaus of Merck and Johnson & Johnson within the past 12 months. No other potential conflicts of interest relevant to this article were reported.

Author Contributions. A.D.M.-H., A.M., T.D.H., S.X.K., and K.M.P. researched and analyzed the data, contributed to the study design/conception, and contributed to the drafting and/or critical revision of the manuscript. A.Z. researched and analyzed the data. X.J., W.W., P.P., and R.S.Z. researched and analyzed the data and contributed to the study design/conception. M.M. researched and analyzed the data and contributed to the drafting and/or critical revision of the manuscript. R.G. contributed to the study design/conception. J.M.B. oversaw the project, contributed to the study design/conception, and contributed to the drafting and/or critical revision of the manuscript. M.W.K. researched and analyzed the data and contributed to the drafting and/or critical revision of the manuscript. All authors have reviewed and approved the final version of the manuscript. A.D.M.-H. is the guarantor of this work and, as such, had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.