We previously developed and validated a hypoglycemia risk stratification tool (1) and made available the computer source code for implementation (online-only supplemental eTable 2 [1]). This tool classifies 12-month risk of hypoglycemia-related utilization (HRU) of emergency department (ED) or in-patient services among type 2 diabetes (T2D) patients as high (>5%), intermediate (1–5%), or low (<1%). Since its publication, health care delivery systems in the U.S. (including Kaiser Permanente and Mayo Clinic) have adopted this tool to identify higher-risk patients for targeted population management interventions designed to reduce hypoglycemia risk. Among the six inputs required to calculate HRU risk (prior HRU, insulin use, sulfonylurea use, any ED visits, chronic kidney disease stage, and age), only prior HRU relied on diagnostic coding and was based on an algorithm (2) comprising International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes. Since 1 October 2015, the Centers for Medicare & Medicaid Services has required use of the 10th revision (ICD-10-CM), necessitating an updated HRU case identification algorithm that includes ICD-10-CM codes for hypoglycemia.

We identified ICD-10-CM codes for hypoglycemia to complement the existing ICD-9-CM–based algorithm (any of the following: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1–707.9, 709.3, 730.0–730.2, or 731.8). The ICD-10-CM codes for hypoglycemia (any of the following: E08.641, E08.649, E09.641, E09.649, E10.641, E10.649, E11.641, E11.649, E13.641, E13.649, E15, E16.0, E16.1, E16.2, T38.3X1A, T38.3X1D, T38.3X1S, T38.3X2A, T38.3X2D, T38.3X2S, T38.3X3A, T38.3X3D, T38.3X3S, T38.3X4A, T38.3X4D, T38.3X4S, T38.3X5A, T38.3X5D, T38.3X5S) are specific to type of diabetes and diabetes status. Although the hypoglycemia risk stratification tool was designed specifically for T2D patients, type of diabetes and even diabetes status may be misclassified in the ED and thus we include all ICD-10-CM codes for hypoglycemia. Using both sets of codes, we tested the performance of the hypoglycemia risk stratification tool among 264,658 active Kaiser Permanente Northern California members, age 21 years or older, diagnosed with T2D as of 1 January 2016 and alive on 1 January 2017 (baseline). We predicted the 12-month risk of HRU (1 January–31 December 2017) and compared it to observed HRU events. HRU events were identified by ICD-9-CM or ICD-10-CM codes depending on whether the event occurred before or after 1 October 2015, respectively. Prebaseline HRU events were used as model inputs, i.e., past HRU events as predictors of future HRU events. HRU events that occurred during the 12-month follow-up after baseline were the outcomes of interest, i.e., what we were predicting. Discrimination, or the tool’s ability to correctly distinguish between subjects who would versus would not experience ≥1 HRU during follow-up (1 January–31 December 2017), was assessed by calculating the area under the receiver operating characteristic curve (C-statistic), with >0.75 classified as good discrimination (3). Clinical utility was assessed by calculating the odds ratio of having ≥1 HRU event during follow-up in those classified as high risk relative to low risk. These performance measures were compared with those observed in the original validation of the hypoglycemia risk stratification tool. This study was approved by the institutional review boards of Kaiser Permanente, the Bedford Veterans Health Administration, and Group Health Cooperative; the requirement that informed consent be obtained from study participants was waived.

The distribution of subjects categorized as high, intermediate, or low HRU risk at baseline (1 January 2017) using the ICD-10-CM updated case identification algorithm closely matched the distribution determined in our original validation, which used only ICD-9 codes with a 1 January 2014 baseline (Table 1). There were similar observed rates of ≥1 HRU events during the 12-month follow-up among subjects classified as having high, intermediate, and low risk in theoriginal and current validation studies, although the rates were somewhat higher in 2017 than in 2014. The incorporation of ICD-10-CM codes did not alter the tool’s ability to discriminate; the C-statistic was identical to that of the original validation (0.83), demonstrating good discrimination. As in the original validation of the tool, there was also excellent clinical utility, with 28-fold greater odds of HRU events occurring during the 12-month follow-up among those categorized as high risk relative to low risk at baseline.

Table 1

Performance of the hypoglycemia risk stratification tool with ICD-9-CM codes for case identification of prior HRU to predict 2014 HRU compared with the performance of the tool using updated case identification (incorporating ICD-10-CM codes) to predict 2017 HRU

Hypoglycemia risk stratification tool with 1 January 2014 baseline (ICD-9-CM case identification only)Hypoglycemia risk stratification tool with 1 January 2017 baseline (updated with ICD-10-CM case identification)
12-Month follow-up (prediction year) 1 January–31 December 2014 1 January–31 December 2017 
Baseline HRU risk categories (%)   
 High 2.0 1.9 
 Intermediate 10.7 11.0 
 Low 87.3 87.1 
Rate of ≥1 HRU observed during 12-month follow-up (%)   
 High risk 6.7 8.8 
 Intermediate risk 1.4 2.2 
 Low risk 0.2 0.3 
Discrimination: area under the receiver operating characteristic curve (C-statistic) 0.83 0.83 
Clinical utility: Odds ratio (95% CI) for ≥1 HRU in prediction year for high- vs. low-risk groups 34.6 (24.2–49.3) 28.0 (24.8–31.5) 
Hypoglycemia risk stratification tool with 1 January 2014 baseline (ICD-9-CM case identification only)Hypoglycemia risk stratification tool with 1 January 2017 baseline (updated with ICD-10-CM case identification)
12-Month follow-up (prediction year) 1 January–31 December 2014 1 January–31 December 2017 
Baseline HRU risk categories (%)   
 High 2.0 1.9 
 Intermediate 10.7 11.0 
 Low 87.3 87.1 
Rate of ≥1 HRU observed during 12-month follow-up (%)   
 High risk 6.7 8.8 
 Intermediate risk 1.4 2.2 
 Low risk 0.2 0.3 
Discrimination: area under the receiver operating characteristic curve (C-statistic) 0.83 0.83 
Clinical utility: Odds ratio (95% CI) for ≥1 HRU in prediction year for high- vs. low-risk groups 34.6 (24.2–49.3) 28.0 (24.8–31.5) 

After updating the case identification algorithm with ICD-10-CM codes, we found that the hypoglycemia risk stratification tool again demonstrated good discrimination and excellent clinical utility in categorizing 12-month risk of HRU of patients with T2D. Use of this hypoglycemia risk stratification tool can facilitate targeting higher-risk patients with population management interventions designed to prevent hypoglycemia (e.g., deprescribing, health education, continuous glucose monitoring, food security) and could potentially improve patient safety and quality of life.

Funding. This project was funded by the U.S. Food and Drug Administration (FDA BAA-13 00119). The National Institutes of Health National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) supplied additional funding for our hypoglycemia-related research (R01 DK103721). A.J.K. and E.S.H. were also supported by NIDDK Centers for Diabetes Translational Research (P30 DK092924 and P30 DK092949, respectively).

The funders were not involved with the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. A.J.K. contributed to study conception and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript for important intellectual content. E.M.W. and H.H.M. contributed to study conception and design, analysis and interpretation of the data, and critical revision of the manuscript for important intellectual content. J.D.R. and D.R.M. contributed to study conception and design, acquisition of the data, and critical revision of the manuscript for important intellectual content. E.S.H. and K.J.L. contributed to study conception and design and critical revision of the manuscript for important intellectual content. A.J.K. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

1.
Karter
AJ
,
Warton
EM
,
Lipska
KJ
, et al
.
Development and validation of a tool to identify patients with type 2 diabetes at high risk of hypoglycemia-related emergency department or hospital use
.
JAMA Intern Med
2017
;
177
:
1461
1470
[PubMed]
2.
Ginde
AA
,
Blanc
PG
,
Lieberman
RM
,
Camargo
CA
 Jr
.
Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits
.
BMC Endocr Disord
2008
;
8
:
4
[PubMed]
3.
Alba
AC
,
Agoritsas
T
,
Walsh
M
, et al
.
Discrimination and calibration of clinical prediction models: users’ guides to the medical literature
.
JAMA
2017
;
318
:
1377
1384
[PubMed]
Readers may use this article as long as the work is properly cited, the use is educational and not for profit, and the work is not altered. More information is available at http://www.diabetesjournals.org/content/license.