Comparison of Diagnostic Accuracy for Lower-Extremity Amputation Codes During the ICD-9 and ICD-10 Eras in a High-Risk Population of Patients With Diabetes

In fiscal year (FY) 2016, the U.S. transitioned to the ICD-10, which includes eight times more amputation procedure codes than the ICD-9. To evaluate trends over time, researchers need assurance that differences over time are not influenced by a change in the coding system. Thus, we aimed to compare the diagnostic accuracy for ascertaining amputations performed during the ICD-9 era compared with during the ICD-10 era, using medical record review as a gold standard. This information will be valuable for others who use electronic health records to study lower-extremity amputation and whose data span this transition.

The source population was veterans who had a procedure code (ICD or Current Procedural Terminology [CPT]) for an initial toe or ray amputation (hereafter referred to as toe amputations) between FY 2005 and FY 2016 and a diagnosis of diabetes or prescription for a diabetes medication in the year prior to their toe amputation (1). We randomly sampled 150 veterans from the parent study (1) who had initial toe amputations in FY 2014 and follow-up through the end of FY 2015 (during the ICD-9 era) and 150 veterans who had initial toe amputations in FY 2016 (during the ICD-10 era). The outcome of interest was an ipsilateral amputation (at any level) in the year after the toe amputation.

Procedure codes were extracted from the Veterans Affairs (VA) Corporate Data Warehouse, which includes data from the VA electronic medical record. To maximize sensitivity for the parent study, we relied on both ICD and CPT codes for identification of amputation procedures (1); there was no change in CPT coding during the study years.

Coders reviewed clinical notes (the gold standard) to find information on amputations from the date of the initial toe amputation to 365 days later. Individuals missing information about the initial toe amputation were excluded (n = 3).

The first author (coder 1) developed the abstraction protocol by reviewing all clinical notes for 23 individuals (12 from the ICD-9 era and 11 from the ICD-10 era) in the year after the initial toe amputation. Coders 2 and 3 first coded 10 records (5 from each time period) that were initially coded by coder 1 to ensure consistent coding. Coder 1 reviewed ∼10% of the remaining records for quality control. To identify amputations, coders searched for note titles including the words “operation,” “operative,” “surgery,” “podiatry,” “procedure,” or “vascular.” If no relevant notes with the above terms were found, all other notes were reviewed for evidence of an amputation or postsurgical healing. Coder 2 reviewed 154 records and coder 3 reviewed 235 records (including 94 records reviewed by both). Of the 64 records reviewed by coder 1, 33 were also reviewed by coder 2 and 49 were also reviewed by coder 3. Coder 1 resolved any differences in coding between coders 2 and 3 and made a final determination. κ, a measure of intercoder agreement, ranged from 0.77 (for coder 1 vs. coder 2) to 0.96 (for coder 1 vs. coder 3).

We assessed diagnostic accuracy in the two eras by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), misclassification, and their respective 95% CIs, according to standard formulas, using chart review as a gold standard.

Based on medical record review, 49 of 149 (32.9%) subjects in the ICD-9 sample and 47 of 148 (31.8%) in the ICD-10 sample had a subsequent ipsilateral amputation. Based on procedure codes, the percentage classified as having a subsequent ipsilateral amputation was slightly higher (51 of 149 [34.2%] in the ICD-9 sample and 56 of 148 [37.8%] in the ICD-10 sample). Compared with the ICD-9 sample, the ICD-10 sample had slightly higher sensitivity (0.83 vs. 0.80), lower specificity (0.83 vs. 0.88), lower PPV (0.70 vs. 0.77), and higher NPV (0.91 vs. 0.90). Misclassification was slightly higher for ICD-10 compared with ICD-9 (0.17 vs. 0.15) (Table 1).

Several factors should be considered when interpreting our findings. First, coders may have misinterpreted information. For example, we classified resection of bone as an amputation, which may not have always been coded (or considered) to meet the threshold definition of an amputation or amputation revision. Second, coders may have missed information about an amputation because of the vast number of notes that some patients had (e.g., >1,000) and procedure information only being included in progress notes or in scanned records for procedures performed outside VA. However, the fraction of amputations performed outside VA was similar during the two time periods (∼14–17%), so it is unlikely that underascertainment of amputation performed outside the VA would explain differences observed. Furthermore, with three coders, we had considerable overlap of review; it is less likely that two or three individuals would all miss or misconstrue important information. Third, coders could not be blinded to the year because it was necessary to have that information to find the relevant records. However, records were alternated for review between ICD-9 and ICD-10 periods, increasing the likelihood that the same approach would be applied consistently regardless of year. Fourth, we used both ICD and CPT procedure codes to identify amputations. Only 16% of subsequent amputations were identified based only on ICD codes; the remainder (84% of amputations) were identified by a combination of CPT and/or ICD codes. This indicates that there was substantial continuity between the two time periods through the use of both ICD and CPT codes. Lastly, it is possible that the diagnostic characteristics differed by characteristics of the patient and/or their amputation (e.g., whether it was performed within or outside VA and whether the subsequent amputation was a minor amputation or a major amputation). Understanding this potential variation was outside the scope of our study, but this would be good to consider as an area for future research.

In conclusion, our study indicates that the sensitivity and specificity of procedure codes for lower-extremity amputation in a population at high risk of amputation were not meaningfully different between the periods when ICD-9 was in use and when ICD-10 came into use. These results support that differences over time in risks of subsequent ipsilateral amputation among patients with an initial toe amputation are not due to underascertainment of procedures in 2014 or overascertainment in 2016.