OBJECTIVE—To assess the performance of health systems using diabetes as a tracer condition.
RESEARCH DESIGN AND METHODS—We generated a measure of “case-fatality” among young people with diabetes using the mortality-to-incidence ratio (M/I ratio) for 29 industrialized countries using published data on diabetes incidence and mortality. Standardized incidence rates for ages 0–14 years were extracted from the World Health Organization DiaMond study for the period 1990–1994; data on death from diabetes for ages 0–39 years were obtained from the World Health Organization mortality database and converted into age-standardized death rates for the period 1994–1998, using the European standard population.
RESULTS—The M/I ratio varied >10-fold. These relative differences appear similar to those observed in cohort studies of mortality among young people with type 1 diabetes in five countries. A sensitivity analysis showed that using plausible assumptions about potential overestimation of diabetes as a cause of death and underestimation of incidence rates in the U.S. yields an M/I ratio that would still be twice as high as in the U.K. or Canada.
CONCLUSIONS—The M/I ratio for diabetes provides a means of differentiating countries on quality of care for people with diabetes. It is solely an indicator of potential problems, a basis for stimulating more detailed assessments of whether such problems exist, and what can be done to address them.
Designing simple, practical, and understandable ways to assess health system performance remains a challenging aspiration. Recognizing the multifunctional complexity of a health system, existing frameworks use a range of indicators to capture the different aspects of health systems (1). However, many of these indicators have no obvious direct link to health outcomes, and the policy implications are often unclear (2). A complementary approach involves the use of tracer conditions (3), which is based on the premise that focusing on carefully selected health problems makes it possible to identify weaknesses in elements of the health system and to obtain more direct insight into its performance.
The rising burden of chronic diseases worldwide demands measures that will capture differences in the care provided to those affected, and we propose diabetes as a suitable tracer condition; it is well defined, fairly easy to diagnose (4), and common. The prevalence worldwide is estimated to be 2.8% (2000) and expected to increase to 4.4% by 2030 (5), with this figure already exceeded in the U.S. (6). While mainly type 2 diabetes is increasing, type 1 diabetes is also increasing swiftly, at ∼3% per year, especially in Central and Eastern Europe and among young children (7).
Health system performance affects diabetes outcomes in several ways. Effective treatment reduces the risk of disabling or fatal complications (8–10). This is most apparent for type 1 diabetes in developing countries where access to insulin is extremely limited (11). However, it is also seen where health systems have collapsed, as in the former Soviet Union (12). Its optimal management requires coordinated inputs from a wide range of health professionals, access to essential medicines and monitoring, and, ideally, a system that promotes patient empowerment. This has relevance beyond diabetes. A health service that is unable to integrate these elements for management of diabetes is unlikely to be able to meet the needs of people with disorders such as asthma, epilepsy, or hypertension. We describe a means of using data on diabetes to make a “preliminary diagnosis” of the effectiveness of diabetes care and which is likely to be generally available to health systems. This will enable the identification of countries where there is a need for more detailed investigation to determine the scale and nature of any problems.
RESEARCH DESIGN AND METHODS—
While many potential measures denote adverse outcomes of diabetes, the only one widely and routinely available is mortality. Mortality comparisons must however take account of diabetes incidence, which varies considerably between countries (13). Building on earlier work (14), we generate a measure of “relative mortality” that combines mortality and incidence, producing a mortality-to-incidence ratio (M/I ratio). The M/I ratio is commonly used in cancer epidemiology as a crude indicator of cancer survival or “case fatality” and thus the overall quality of health care.
Underrecording of diabetes on death certificates is well documented (15), particularly among the elderly, although certification at younger ages, usually with type 1 diabetes, is more reliable (16). Thus, to maximize diagnostic specificity, comparisons should be restricted to deaths at young ages. However, such deaths from diabetes are few, so the choice of an upper age limit requires a trade off between including a greater number of deaths from type 2 diabetes against the need to minimize random fluctuation due to small numbers. After inspecting the crude data over several years to assess the extent of variation with different cutoff points, we chose 39 years as the upper age limit for our primary analyses.
Mortality and population data were extracted from the World Health Organization mortality database (17). Data include deaths by sex, 5-year age band, and cause according to the ICD. We examined data for 29 countries for which uniform incidence data were available (see below). We calculated age-standardized death rates from diabetes per 100,000 population for the age-group 0–39 years (both sexes combined), using the European standard population (18). Death from diabetes was classified according to ICD 9th (250) or 10th revision (E10–E14). Here, we use a 5-year average over the period 1994–1998, the latest period for which data were available for all 29 countries. Exceptions were Germany, Canada, Israel, and Poland, where data were only available until 1997 or 1996 (Poland). Here, we used a 4- and 3-year average, respectively.
Published age-standardized incidence rates among children aged 0–14 years (per 100,000 population) were extracted from the World Health Organization DiaMond study, which collected standardized incidence data on type 1 diabetes for the period 1990–1994 using population registries in 50 countries (13). For Germany, Italy, and Latvia, we used published data from the EURODIAB study (19), which appeared to be more nationally representative. Where two or more centers in a country reported independently, we calculated a simple national average without weighting for sample size or regional sample fraction (which was unknown). We computed the M/I ratio for 29 countries, using the age-standardized death rate from diabetes for ages 0–39 years (1994–1998) and the published age-standardized incidence rate for ages 0–14 years (1990–1994) as described above.
Sensitivity analysis
To assess the robustness of the M/I ratio, we focused on the U.S., which had the second-highest M/I ratio among Western industrialized countries (Table 1). We developed a series of scenarios that assessed possible biases that might falsely elevate the U.S. M/I ratio: 1) overassignment of diabetes as an underlying cause of death in the U.S. (by 10, 20, and 50%), 2) underestimation of the “true” national incidence rate by using the average of three regional incidence rates, replacing this average by the highest regional incidence or the upper bound of the 95% CI, and 3) a combination of 1) and 2).
Validation
To externally validate our findings, we also present specific survival data from cohorts of subjects with diabetes from countries included in our study, as published by the Diabetes Epidemiology Research International Study of mortality among young people (aged <40 years) with insulin-dependent diabetes, followed until 1990, for Israel, Finland, the U.S., and Japan (20) and from a cohort study from the U.K. of age-specific mortality rates among insulin-treated patients aged 30–39 years (21).
RESULTS—
Figure 1 shows a 20-fold variation in the age-standardized incidence rates across the 29 countries included in our analysis, from a low of <2 cases annually per 100,000 population in Japan to >35 per 100,000 population in Finland. Substantial independent variation in mortality is also apparent, with the pattern very different from that of incidence, ranging between 0.14 deaths per 100,000 population per year in Greece (medium incidence) and levels >10 times that in Russia (very low incidence), Estonia (medium incidence), and Finland (highest incidence).
Converting these data to a M/I ratio (Fig. 2), we find a large variation, with highest ratios in former communist countries of Central and Eastern Europe, along with Japan and the U.S. Individual cohort data (Fig. 2, insert) demonstrates mortality among young people (aged <40 years) with insulin-dependent diabetes, followed until 1990, to be much lower in Israel and Finland than in the U.S. or Japan, at 158 and 250 per 100,000 person-years compared with 408 and 760/100,000 person-years, respectively. The ratios of the rates for those aged 30–39 years in these cohorts to that from the U.K. cohort (1.3, 1.7, 3.9, and 5.2) are fairly similar to those obtained by comparing their national M/I ratios (aged 0–39 years) to the U.K. M/I ratio (1.7, 1.7, 3.3, and 7.8). This may be interpreted as an additional external validation of our approach.
The sensitivity analyses (Table 1) indicate that even assuming a relative overassignment of diabetes as a cause of death among those aged 0–39 years in the U.S. of 20% and a high incidence rate of type 1 diabetes among 0- to 14-year-old children of 17.8/100,000 population (as reported for Allegheny County, PA), the M/I ratio would still be twice as high in the U.S. as in the U.K. or Canada.
CONCLUSIONS—
We present a measure that aims to capture differences in health system performance using (routinely) available data. We find a remarkable variation across countries, suggesting gross differences in the ability of health systems to provide adequate care for people with diabetes. It seems very likely that those countries with high values of the M/I ratio (which may be defined to be >0.1) are not performing well in delivering effective care for diabetes and that as a group they are distinctly different from those with low values (which may be defined to be <0.025). The intermediate group spans a fair range, and their “true” relative performance, especially toward the lower end of the scale, is less obvious. However, our sensitivity analysis suggests that the difference between a country toward the upper end and the low group is probably not artifactual.
Of course, the composite nature of the M/I ratio, using data from (mostly) different individuals spread across different mortality cohorts, means that no firm conclusions can be drawn solely from these figures, a constraint exacerbated by the variability resulting from small numbers of deaths in some countries. Indeed, producing a ranking as such is not the purpose of a tracer condition. Rather, the relative position of a country is to be interpreted as a warning flag signaling the need for a more in-depth assessment and a tool for monitoring progress in the longer term.
There are obvious potential uncertainties about the index used in our analysis. Even using a high maximum age of 39 years, some countries accumulated <50 deaths from diabetes over 5 years; but very large numbers of deaths in some widely separated anchor points suggest that chance variation is not a prime explanation for this spread (e.g., 1,045 deaths in Russia, 588 in Japan, 7,628 in the U.S., 607 in the U.K., and 469 in Italy). We are not able to exclude differentials in diagnosis and coding across countries, although this problem is at least partially addressed by the sensitivity analyses. There is good evidence that for at least some of the countries at opposite ends of the M/I scale coding practices are rather similar (15), and other work (14) noted similar findings for data based on deaths at ages 0–24 years during the 1980s. In addition, comparisons with survival data from prospective cohorts are congruent with our population-level observations (Fig. 2), providing an important external validation of our approach.
Next steps: the system response to a “positive” trace
The identification of a problem using a health system’s M/I ratio is only the first step. The next step involves examining complementary data to understand the immediate causes of death driving the differences and links to possible underlying organizational and system weaknesses or failures. Thus, much of the excess mortality in Japan was found to be due to diabetic renal disease (22), largely caused by higher incidence of end-stage renal disease and reduced access to dialysis compared with the U.S. (23). Lower survival rates among those with type 1 diabetes in Estonia and Latvia compared with Finland are explained by a higher proportion of deaths due to acute complications of diabetes (24), reflecting, among others, the greater experience in Finland where type 1 diabetes is much more common than in the Baltic states.
The U.S. has come under particular scrutiny because of their relatively poor outcomes (which are captured by our index), and available evidence suggests that organizational and social factors may be more prominent than clinical ones (25,26). It is also important to note that disadvantaged population subgroups may contribute to higher mortality (27), and in the U.S. complications and mortality from diabetes are known to be worst among the population of African descent (28,29). Yet, while the relative contributions of etiology, health care, and non–health care determinants to diabetes outcome are still inadequately understood (30), health care can play a substantial part in alleviating the health impact of disadvantage due to socioeconomic status (27,31) or ethnicity (32).
We fully recognize the limitations of measures such as the M/I ratio and emphasize that they are only indicators that should prompt more detailed investigation as noted above. It is, however, important not to lose sight of the initial premise underlying this work. Diabetes is both important in its own right and a tracer disease that can provide insights into the ability of the health system to respond to chronic disorders. The key elements of an effective response to each disorder are similar, including continuity of care; integration of care across primary, secondary, and social care interfaces; and active patient involvement. This is supported by the observation that the U.S. performs considerably worse than other industrialized countries, as assessed by premature deaths from not only diabetes but also hypertension and obstructive airways disease (33). However, comparisons of the care provided for these disorders pose even greater problems than diabetes when using routinely available data.
Age-standardized death rates (SDR) from diabetes at ages 0–39 (1994–1998) years and age-standardized incidence rates (SIR) at ages 0–14 (1990–1994) years in 29 countries.
Age-standardized death rates (SDR) from diabetes at ages 0–39 (1994–1998) years and age-standardized incidence rates (SIR) at ages 0–14 (1990–1994) years in 29 countries.
M/I ratio (SDR:SIR) for diabetes in 29 countries. Insert: mortality among people (<40 years) with insulin-dependent diabetes in the Diabetes Epidemiology Research International and the British Diabetic Association cohorts (20,21).
Sensitivity analyses of the ratio of national standardized death rates and the ratio of M/I ratios
Scenarios . | Ratio of national standardized death rate (1994–1998) . | . | Ratio of M/I ratios . | . | ||
---|---|---|---|---|---|---|
. | U.S. vs. U.K. . | U.S. vs. Canada . | vs. U.K. . | vs. Canada . | ||
Scenario 1: as reported (U.S. incidence: 14.8/100,000 population) | 2.6 | 2.0 | 3.3 | 3.2 | ||
Scenario 2: U.S. death rate | ||||||
Excess: 10% | 2.4 | 1.8 | 3.0 | 2.9 | ||
Excess: 20% | 2.1 | 1.6 | 2.7 | 2.6 | ||
Excess: 50% | 1.3 | 1.0 | 1.7 | 1.6 | ||
Scenario 3: increase U.S. incidence rate to | ||||||
a) highest regional rate (17.8/100,000 population) (13) | — | — | 2.8 | 2.7 | ||
b) upper 95% CI of highest regional rate (20.3/100,000 population) (13) | — | — | 2.4 | 2.4 | ||
Scenario 4: scenario 3a + 20% mortality excess | — | — | 1.9 | 1.9 |
Scenarios . | Ratio of national standardized death rate (1994–1998) . | . | Ratio of M/I ratios . | . | ||
---|---|---|---|---|---|---|
. | U.S. vs. U.K. . | U.S. vs. Canada . | vs. U.K. . | vs. Canada . | ||
Scenario 1: as reported (U.S. incidence: 14.8/100,000 population) | 2.6 | 2.0 | 3.3 | 3.2 | ||
Scenario 2: U.S. death rate | ||||||
Excess: 10% | 2.4 | 1.8 | 3.0 | 2.9 | ||
Excess: 20% | 2.1 | 1.6 | 2.7 | 2.6 | ||
Excess: 50% | 1.3 | 1.0 | 1.7 | 1.6 | ||
Scenario 3: increase U.S. incidence rate to | ||||||
a) highest regional rate (17.8/100,000 population) (13) | — | — | 2.8 | 2.7 | ||
b) upper 95% CI of highest regional rate (20.3/100,000 population) (13) | — | — | 2.4 | 2.4 | ||
Scenario 4: scenario 3a + 20% mortality excess | — | — | 1.9 | 1.9 |
Scenario 1, no assumption; scenario 2, corrected for assumed overestimation of diabetes mortality in the U.S. by 10, 20, or 50% due to overassignment of diabetes as underlying cause of death; scenario 3, corrected for assumed underestimation of true national incidence rate through averaging regional rates: increase incidence rate to highest regional rate (scenario 3a) or the upper 95% CI of highest regional rate (scenario 3b); and scenario 4, corrected for combination of overestimation of mortality and underestimation of incidence rate.
Article Information
E.N. is the holder of a fellowship on international benchmarking awarded by the Nuffield Trust. C.B. was supported in part by a Leverhulme Visiting Fellowship from the University of Bristol. M.M.’s work on health systems performance is funded through the Health Systems Development Knowledge Program of the U.K. Department for International Development. However, none of the sponsors can accept responsibility for the views expressed.
References
A table elsewhere in this issue shows conventional and Système International (SI) units and conversion factors for many substances.
The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked “advertisement” in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.