Diabetes as a Tracer Condition in International Benchmarking of Health Systems

While many potential measures denote adverse outcomes of diabetes, the only one widely and routinely available is mortality. Mortality comparisons must however take account of diabetes incidence, which varies considerably between countries (13). Building on earlier work (14), we generate a measure of “relative mortality” that combines mortality and incidence, producing a mortality-to-incidence ratio (M/I ratio). The M/I ratio is commonly used in cancer epidemiology as a crude indicator of cancer survival or “case fatality” and thus the overall quality of health care.

Underrecording of diabetes on death certificates is well documented (15), particularly among the elderly, although certification at younger ages, usually with type 1 diabetes, is more reliable (16). Thus, to maximize diagnostic specificity, comparisons should be restricted to deaths at young ages. However, such deaths from diabetes are few, so the choice of an upper age limit requires a trade off between including a greater number of deaths from type 2 diabetes against the need to minimize random fluctuation due to small numbers. After inspecting the crude data over several years to assess the extent of variation with different cutoff points, we chose 39 years as the upper age limit for our primary analyses.

Mortality and population data were extracted from the World Health Organization mortality database (17). Data include deaths by sex, 5-year age band, and cause according to the ICD. We examined data for 29 countries for which uniform incidence data were available (see below). We calculated age-standardized death rates from diabetes per 100,000 population for the age-group 0–39 years (both sexes combined), using the European standard population (18). Death from diabetes was classified according to ICD 9th (250) or 10th revision (E10–E14). Here, we use a 5-year average over the period 1994–1998, the latest period for which data were available for all 29 countries. Exceptions were Germany, Canada, Israel, and Poland, where data were only available until 1997 or 1996 (Poland). Here, we used a 4- and 3-year average, respectively.

Published age-standardized incidence rates among children aged 0–14 years (per 100,000 population) were extracted from the World Health Organization DiaMond study, which collected standardized incidence data on type 1 diabetes for the period 1990–1994 using population registries in 50 countries (13). For Germany, Italy, and Latvia, we used published data from the EURODIAB study (19), which appeared to be more nationally representative. Where two or more centers in a country reported independently, we calculated a simple national average without weighting for sample size or regional sample fraction (which was unknown). We computed the M/I ratio for 29 countries, using the age-standardized death rate from diabetes for ages 0–39 years (1994–1998) and the published age-standardized incidence rate for ages 0–14 years (1990–1994) as described above.

To assess the robustness of the M/I ratio, we focused on the U.S., which had the second-highest M/I ratio among Western industrialized countries (Table 1). We developed a series of scenarios that assessed possible biases that might falsely elevate the U.S. M/I ratio: 1) overassignment of diabetes as an underlying cause of death in the U.S. (by 10, 20, and 50%), 2) underestimation of the “true” national incidence rate by using the average of three regional incidence rates, replacing this average by the highest regional incidence or the upper bound of the 95% CI, and 3) a combination of 1) and 2).

To externally validate our findings, we also present specific survival data from cohorts of subjects with diabetes from countries included in our study, as published by the Diabetes Epidemiology Research International Study of mortality among young people (aged <40 years) with insulin-dependent diabetes, followed until 1990, for Israel, Finland, the U.S., and Japan (20) and from a cohort study from the U.K. of age-specific mortality rates among insulin-treated patients aged 30–39 years (21).

Figure 1 shows a 20-fold variation in the age-standardized incidence rates across the 29 countries included in our analysis, from a low of <2 cases annually per 100,000 population in Japan to >35 per 100,000 population in Finland. Substantial independent variation in mortality is also apparent, with the pattern very different from that of incidence, ranging between 0.14 deaths per 100,000 population per year in Greece (medium incidence) and levels >10 times that in Russia (very low incidence), Estonia (medium incidence), and Finland (highest incidence).

Converting these data to a M/I ratio (Fig. 2), we find a large variation, with highest ratios in former communist countries of Central and Eastern Europe, along with Japan and the U.S. Individual cohort data (Fig. 2, insert) demonstrates mortality among young people (aged <40 years) with insulin-dependent diabetes, followed until 1990, to be much lower in Israel and Finland than in the U.S. or Japan, at 158 and 250 per 100,000 person-years compared with 408 and 760/100,000 person-years, respectively. The ratios of the rates for those aged 30–39 years in these cohorts to that from the U.K. cohort (1.3, 1.7, 3.9, and 5.2) are fairly similar to those obtained by comparing their national M/I ratios (aged 0–39 years) to the U.K. M/I ratio (1.7, 1.7, 3.3, and 7.8). This may be interpreted as an additional external validation of our approach.

The sensitivity analyses (Table 1) indicate that even assuming a relative overassignment of diabetes as a cause of death among those aged 0–39 years in the U.S. of 20% and a high incidence rate of type 1 diabetes among 0- to 14-year-old children of 17.8/100,000 population (as reported for Allegheny County, PA), the M/I ratio would still be twice as high in the U.S. as in the U.K. or Canada.

We present a measure that aims to capture differences in health system performance using (routinely) available data. We find a remarkable variation across countries, suggesting gross differences in the ability of health systems to provide adequate care for people with diabetes. It seems very likely that those countries with high values of the M/I ratio (which may be defined to be >0.1) are not performing well in delivering effective care for diabetes and that as a group they are distinctly different from those with low values (which may be defined to be <0.025). The intermediate group spans a fair range, and their “true” relative performance, especially toward the lower end of the scale, is less obvious. However, our sensitivity analysis suggests that the difference between a country toward the upper end and the low group is probably not artifactual.

Of course, the composite nature of the M/I ratio, using data from (mostly) different individuals spread across different mortality cohorts, means that no firm conclusions can be drawn solely from these figures, a constraint exacerbated by the variability resulting from small numbers of deaths in some countries. Indeed, producing a ranking as such is not the purpose of a tracer condition. Rather, the relative position of a country is to be interpreted as a warning flag signaling the need for a more in-depth assessment and a tool for monitoring progress in the longer term.

There are obvious potential uncertainties about the index used in our analysis. Even using a high maximum age of 39 years, some countries accumulated <50 deaths from diabetes over 5 years; but very large numbers of deaths in some widely separated anchor points suggest that chance variation is not a prime explanation for this spread (e.g., 1,045 deaths in Russia, 588 in Japan, 7,628 in the U.S., 607 in the U.K., and 469 in Italy). We are not able to exclude differentials in diagnosis and coding across countries, although this problem is at least partially addressed by the sensitivity analyses. There is good evidence that for at least some of the countries at opposite ends of the M/I scale coding practices are rather similar (15), and other work (14) noted similar findings for data based on deaths at ages 0–24 years during the 1980s. In addition, comparisons with survival data from prospective cohorts are congruent with our population-level observations (Fig. 2), providing an important external validation of our approach.

The identification of a problem using a health system’s M/I ratio is only the first step. The next step involves examining complementary data to understand the immediate causes of death driving the differences and links to possible underlying organizational and system weaknesses or failures. Thus, much of the excess mortality in Japan was found to be due to diabetic renal disease (22), largely caused by higher incidence of end-stage renal disease and reduced access to dialysis compared with the U.S. (23). Lower survival rates among those with type 1 diabetes in Estonia and Latvia compared with Finland are explained by a higher proportion of deaths due to acute complications of diabetes (24), reflecting, among others, the greater experience in Finland where type 1 diabetes is much more common than in the Baltic states.

The U.S. has come under particular scrutiny because of their relatively poor outcomes (which are captured by our index), and available evidence suggests that organizational and social factors may be more prominent than clinical ones (25,26). It is also important to note that disadvantaged population subgroups may contribute to higher mortality (27), and in the U.S. complications and mortality from diabetes are known to be worst among the population of African descent (28,29). Yet, while the relative contributions of etiology, health care, and non–health care determinants to diabetes outcome are still inadequately understood (30), health care can play a substantial part in alleviating the health impact of disadvantage due to socioeconomic status (27,31) or ethnicity (32).

We fully recognize the limitations of measures such as the M/I ratio and emphasize that they are only indicators that should prompt more detailed investigation as noted above. It is, however, important not to lose sight of the initial premise underlying this work. Diabetes is both important in its own right and a tracer disease that can provide insights into the ability of the health system to respond to chronic disorders. The key elements of an effective response to each disorder are similar, including continuity of care; integration of care across primary, secondary, and social care interfaces; and active patient involvement. This is supported by the observation that the U.S. performs considerably worse than other industrialized countries, as assessed by premature deaths from not only diabetes but also hypertension and obstructive airways disease (33). However, comparisons of the care provided for these disorders pose even greater problems than diabetes when using routinely available data.

E.N. is the holder of a fellowship on international benchmarking awarded by the Nuffield Trust. C.B. was supported in part by a Leverhulme Visiting Fellowship from the University of Bristol. M.M.’s work on health systems performance is funded through the Health Systems Development Knowledge Program of the U.K. Department for International Development. However, none of the sponsors can accept responsibility for the views expressed.