Computer Modeling of Diabetes and Its Complications: A report on the Fourth Mount Hood Challenge Meeting

In July 2004, participating modelers received instructions outlining the Fourth Mount Hood Challenge. The challenge had three parts: simulation of a trial of type 2 diabetes (Collaborative Atorvastatin Diabetes Study [CARDS]) that had not been used by any of the modelers to build their model, simulation of a trial of type 1 diabetes (Diabetes Control and Complications Trial [DCCT]) that had been used by the three groups who model type 1 diabetes to build models of that disease; and estimation of outcomes for a precisely defined hypothetical person with type 2 diabetes, with and without glycemic control. This article reports the results of the first and second challenges. Results of simulating the third challenge are described on the Mount Hood 4 Web site (13).

The first part of the challenge was to simulate the outcomes of CARDS, which investigated the role of lipid-lowering therapy in the primary prevention of cardiovascular disease (CVD) in patients with type 2 diabetes. This challenge was undertaken to compare modeling results in a well-defined cohort receiving an intervention to control the progression of cardiovascular complications of type 2 diabetes. CARDS was a randomized placebo-controlled trial that compared 10 mg atorvastatin with placebo in 2,838 patients aged 40–75 years with no clinical history of coronary, cerebrovascular, or severe peripheral vascular disease (LDL cholesterol ≤4.14 mmol/l [≤160 mg/dl]; triglycerides ≤6.78 mmol/l [≤600 mg/dl]) and at least one of the following: hypertension (systolic blood pressure >140 mmHg or diastolic blood pressure >80 mmHg or taking antihypertensive medication), retinopathy, micro- or macroalbuminuria, or current smoker (14). The combined primary end point comprised acute coronary heart disease (CHD) death, nonfatal myocardial infarction (MI) including silent myocardial infarction, hospitalized unstable angina, resuscitated cardiac arrest, coronary revascularization, and stroke.

Results from CARDS were first reported at the ADA 64th Annual Scientific Sessions and posted on the CARDS Web site (http://www.cardstrial.org) on 3 June 2004 (15). For the challenge, modelers (except Archimedes, see below) were sent information on 22 July 2004 by the Mount Hood 4 Steering Group on the design of the trial, baseline characteristics of people in the two groups, and, based on the ADA presentation, the effects of treatment on the most important biological outcomes over the duration of the trial, as well as the results for the primary and secondary end points (14). Archimedes performed its calculations before any postrandomization results were known. The findings of CARDS were subsequently published by Colhoun et al. (16). Modelers were challenged to simulate the trial outcomes using information on the baseline cohort characteristics and changes in risk factor values (e.g., HDL cholesterol) in the treatment and control groups and to report the cumulative incidence of cardiovascular events, all-cause mortality, and quality-adjusted life expectancy taking into account macrovascular complications and the costs of complications by therapy allocation over time horizons of 4 years (trial duration 3.9 years) and 20 years. Each modeling group was requested to use only the source information indicated in order to standardize the inputs for each model (as far as was possible with the different models' structures and designs). Each group was instructed to carefully record the assumptions used when defining the simulations. More information on the simulation settings is provided online at the Mount Hood Challenge 4 Web site (13).

Although modelers were asked to simulate all complications of diabetes over 4- and 20-year time horizons, this article only presents data that can be compared with the published CARDS results with the primary and secondary end points at 4 years. The interested reader is referred to the online documentation from the meeting for results on other complications and health economic outcomes and for the 20-year horizon (13).

The second part of the Mount Hood Challenge was to simulate the outcomes of the DCCT. Modelers were asked to simulate the DCCT type 1 diabetes interventions in the primary prevention cohort and the combined cohort (primary prevention and secondary interventions groups) over time horizons of 9 (trial duration) and 20 years (17–24). Simulations were designed to compare conventional with intensive insulin therapy and were based on published DCCT data. Modelers were given various data from the trial to form the basis of their simulations. This included the observations that A1C levels remained constant in the conventional group but decreased by 1.9% points in the intensive therapy group; that separation in A1C curves was evident after 6 months and was maintained over the long term; that major hypoglycemic event rates were 19 per 100 patient-years on conventional therapy and 62 per 100 patient-years in the intensive therapy group; and that body weight increased by 4.6 kg more in the intensive therapy group than in the conventional group. Additional detail on the assumptions made for these simulations is provided on the Mount Hood 4 Web site (13). In the present article, we report only selected outcomes for microvascular complications that can be compared against published DCCT data.

The format of the Fourth Mount Hood Challenge Meeting was decided in August 2003 at the Third Mount Hood Challenge. Eight modeling groups participated in Third Mount Hood Challenge: the Cardiff Diabetes Model, the Sheffield Diabetes Model, the U.K. Prospective Diabetes Study (UKPDS) Model, the Economic Assessment of Glycemic Control and Long-term Effects (EAGLE) Model, the Center for Outcomes Research (CORE) Diabetes Model, the Archimedes Model, the Global Diabetes Model, the University of Michigan Model, and the Diabetes Decision Analysis and Cost of Type 2 (DiDACT) Model. An open invitation to all diabetes modelers was extended to participate in the Fourth Mount Hood Challenge. Seven models participated in the fourth challenge: the Cardiff Diabetes Model, the Sheffield Diabetes Model, the UKPDS Outcomes Model, the UKPDS Risk Engine, the EAGLE Model, the CORE Diabetes Model, and the CDC/RTI (Centers for Disease Control/Research Triangle Institute) Type 2 Diabetes Progression Model. An eighth model presented data from previous analyses (the Archimedes Model).

The Cardiff Diabetes Model was presented for the first time at the Fourth Mount Hood Challenge Meeting and is a discrete event model that runs stochastic simulations. It was programmed in Visual Basic script embedded in Microsoft Excel and is structurally based on the Eastman Model of type 2 diabetes first published in 1997 (22). It is designed to evaluate the impact of new therapies typically in simulated populations of up to 10,000 newly diagnosed type 2 diabetic patients. The model can accommodate type 2 diabetic and nondiabetic populations using the UKPDS Risk Engine formula (3–5) and the Framingham risk equation, respectively. Standard outputs from model simulations include the incidence of microvascular (retinopathy, neuropathy, or nephropathy) and macrovascular complications (congestive heart disease, MI, sudden death, and cerebrovascular disease defined as neurological deficit with symptoms or signs lasting ≥1 month), mortality, cost-effectiveness and cost-utility data, and acceptability curves.

Source data for the model have been taken from DCCT and UKPDS report 64 for nephropathy, and from UKPDS reports 56 and 60 for macrovascular complications (3,4,21,25). Risk factors included in the simulations (dictated by the risk formula used) include total cholesterol, HDL cholesterol, A1C, systolic blood pressure, smoking status, age, sex, ethnicity, and duration of diabetes. The Cardiff Diabetes Model includes health utility data (using the EQ-5D and the SF-36 instruments) drawn from the Health Outcomes Data Repository database in Cardiff that includes data on 40,000 primary and secondary care patients, of whom 3,000 have diabetes. As the model is not designed to simulate the progression of type 1 diabetes, no data for the DCCT challenge were presented at Mount Hood Challenge 4.

Work on the Sheffield Diabetes Model began in the School of Health and Related Research at the University of Sheffield, Sheffield, U.K., at the end of 2003, and the model was presented at the Fourth Mount Hood Challenge, even though it was still considered to be under development. This model is designed to simulate the progression of type 2 diabetes and consists of five submodels representing comorbidity classifications (CHD, stroke, nephropathy, retinopathy, and neuropathy). The effects of therapy are simulated on the progression of the three main risk factors—-A1C, lipids (total cholesterol and HDL), and blood pressure. Smoking status, age, sex, ethnicity, and duration of diabetes are also taken into consideration in the CVD (CHD and stroke) submodels, as these are primarily based on the UKPDS Risk Engine (UKPDS reports 56 and 60) (3,4).

For the CHD and stroke submodels, source data were taken from various UKPDS publications (3–5,26,27), British Heart Foundation statistics, the PRAIS-U.K. (Prospective Registry of Acute Ischemic Syndromes in the U.K.) study, Minnesota population and NICE (National Institute for Clinical Excellence) guidelines, the PROGRESS (Perindopril Protection Against Recurrent Stroke Study) trial, and the Heart Protection Study. The nephropathy submodel is largely based on published data from the Eastman Model of diabetes progression, with additional data from the Rochester Epidemiology Project and the DCCT (22). Similarly, the retinopathy and neuropathy submodels are based on the Eastman Model, with the former including additional data from the UKPDS 50 report (22,28). The Sheffield Diabetes Model was used to perform the CARDS challenge at Mount Hood Challenge 4, but as this model is designed only to simulate the progression of type 2 diabetes, no results for the DCCT challenge are presented.

The UKPDS Outcomes Model was developed at the University of Oxford on the basis of survival equations estimated using 3,642 patients from the UKPDS, and an article providing a full description of the model was published in Diabetologia (2). In summary, the UKPDS Outcomes Model is a probabilistic discrete-time model that uses an integrated set of parametric proportional hazard models to predict absolute risk of first occurrence of seven major diabetes-related complications (ischemic heart disease, MI [fatal and nonfatal], heart failure, stroke, blindness, renal failure, and amputation). These predictions are based on patient characteristics (such as age and sex) and risk factors that vary with time (such as A1C level and systolic blood pressure). Simulated patients start with a prespecified health status and can experience one or more nonfatal complications or die during any of the annual cycles as the simulation progresses. As a model of type 2 diabetes, the UKPDS Outcomes Model was used to simulate the CARDS challenge at the Mount Hood Challenge 4 but did not perform the DCCT challenge. The UKPDS Outcomes Model is available as software from the UKPDS Web site (http://www.dtu.ox.ac.uk/).

For the CARDS challenge, the UKPDS Outcomes Model (and the UKPDS Risk Engine [see below]) used a simulated cohort with mean risk factors, male-to-female ratio, smoking prevalence, and ethnic mix exactly equal to those reported for CARDS, with risk factor variances approximately equal to those reported for CARDS, and with correlations between risk factors based on the UKPDS.

The UKPDS Risk Engine is a coronary risk calculator for clinical use. It is derived from UKPDS data but designed to reflect risk in the general type 2 diabetic population, not to reproduce the UKPDS (3). UKPDS Risk Engine software provides a diabetes-specific alternative to the many existing coronary risk tools for the general population. The UKPDS Risk Engine equations have also been published (3–5), in part so that they could be used in developing simulation models for health decision making. Although the UKPDS group has subsequently published their own simulation model (see UKPDS Outcomes Model above), other models use UKPDS Risk Engine equations in this way, including several reported here.

The Risk Engine equations available at the time of the meeting estimated the risk of CHD, stroke, fatal CHD, and fatal stroke. (This corresponds to version 2 of the software; version 1 lacked fatal CHD and fatal stroke, but otherwise gave identical results.) These currently available risk equations use risk factors measured in the first 2 years after diagnosis of diabetes. Supporting evidence for their use in cohorts with recently diagnosed type 2 diabetes is available elsewhere (see the results and discussion section). The CARDS challenge tested whether they could also be used in a cohort with several years since diagnosis of diabetes. The forthcoming cardiovascular version of the UKPDS Risk Engine, which is designed for use at any time after diagnosis of diabetes, was not available at the time of the meeting (29). Like the Framingham equations, the UKPDS Risk Engine reports risk from censored survival models. It was incorporated into a simple simulation model for the CARDS challenge. The UKPDS Risk Engine is specific to type 2 diabetes and did not participate in the DCCT challenge.

The EAGLE Model was presented at the Fourth Mount Hood Challenge Meeting by Analytica International, Lörrach, Germany. The EAGLE Model has been outlined at previous Mount Hood Challenge Meetings and is an object-oriented probabilistic Monte Carlo simulation application (30,31). A description of an updated version of the model (version 2.0) that captures the effects of more parameters/variables has been recently published (32). A Markov process with yearly intervals and first- and second-order calculations was the basic structure for development. Transition probabilities are dependent on the status of the simulated patient, with related calculations defined internally. Among various demographics (e.g., age, duration of diabetes, and sex), physiologic characteristics (e.g., A1C and systolic blood pressure), preexisting complications, and lifestyle input parameters (e.g., smoking), the main determinant of events is A1C, which is simulated over time with regard to predefined target A1C. Twenty outcomes (e.g., hypoglycemia, retinopathy, macular edema, end-stage renal disease, neuropathy, diabetic foot syndrome, MI, and stroke) are projected based on data from epidemiological and clinical trials including the DCCT, UKPDS, and WESDR (Wisconsin Epidemiological Study of Diabetic Retinopathy). It is capable of simulating the progression of type 1 and type 2 diabetes. The model generates a virtual patient cohort of up to 50,000 patients for each simulation applying distribution assumptions to determine patient characteristics. Events are assigned to individual patients at the time of occurrence. Results are reported as average values for each cohort simulation and as average values for multiple iterations (of identical simulations) as required. The model also has a health economics module to provide output data on costs (of treatment and complications), cost-consequence, quality of life, and cost-effectiveness of interventions. The EAGLE Model version 2.0, with many more influence parameters and features, has been developed (documentation can be obtained upon request from Analytica). EAGLE Model version 1.2 was used to perform the simulations for the CARDS and DCCT challenges at Mount Hood Challenge 4.

The CORE Diabetes Model was developed by the Center for Outcomes Research, Basel, Switzerland, and has been previously described in detail (7). It is an interactive computer model developed to determine the long-term health outcomes and economic consequences of interventions in type 1 or type 2 diabetes. The model structure comprises 14 interdependent submodels that simulate the complications of diabetes (angina, MI, congestive heart failure, stroke, peripheral vascular disease, diabetic retinopathy, macula edema, cataract, hypoglycemia, ketoacidosis, lactic acidosis, nephropathy and end-stage renal disease, neuropathy, foot ulcer, and amputation) and another simulating nonspecific mortality. Each submodel is a Markov model using time, state, and diabetes type–dependent probabilities. Monte Carlo simulation using tracker variables overcomes the memory-less properties of the standard Markov model and allows interconnectivity and interaction between individual complication submodels. Separate transition probabilities and management strategies are used for type 1 and type 2 diabetes wherever differential data exist. Source data for the model have been taken from a broad range of published clinical and epidemiological studies (7). The economics module allows estimation of direct and indirect costs, applies quality-of-life utilities/disutilities, and allows users to perform cost-effectiveness, cost-utility, budget impact, and cost-benefit analyses. The model has been validated against epidemiological and clinical studies of type 1 diabetes (8). The CORE Diabetes Model was used to simulate the CARDS and DCCT challenges at the Mount Hood Challenge 4 Meeting.

The CDC/RTI Type 2 Diabetes Progression Model was developed by the CDC and RTI International. The model was designed to assess interventions important to public health and has been presented at previous Mount Hood Meetings and outlined in several recent publications (9,10,33,34). Full details are in the Technical Report for the CDC/RTI Type 2 Diabetes Progression Model available directly from the CDC. The model is a Markov simulation programmed in C++ that begins at the diagnosis of type 2 diabetes and tracks a cohort until death. It is based on data from the UKPDS and several other published sources. Patients progress simultaneously through five different disease paths in the model to simulate the complications of diabetes (nephropathy, neuropathy, retinopathy, CHD, and stroke). Transition probabilities for progression through these paths are dependent on time of diagnosis of type 2 diabetes, age, sex, ethnic group, A1C level, smoking status, serum cholesterol levels, and blood pressure.

For reporting clinical outcomes and mortality, the CDC/RTI Type 2 Diabetes Progression Model includes a health economics module that allows calculation of incremental cost-effectiveness ratios for selected interventions. Cohorts can be defined by age, sex, race, hypertension, cholesterol, and smoking status. Updates since the Third Mount Hood Meeting in Oxford, U.K., include incorporation of the UKPDS Risk Engine for simulation of CVD, a facility to simulate screening for type 2 diabetes and pre-diabetic states, and inclusion of probabilistic sensitivity analyses. As the CDC/RTI Type 2 Diabetes Progression Model was not designed to simulate type 1 diabetes, only the CARDS challenge simulations were performed for the Fourth Mount Hood Challenge.

Another model represented at the Fourth Mount Hood Challenge was the Archimedes Model. Developed at Kaiser Permanente, the Archimedes Model has been presented at previous Mount Hood Meetings and described in a recent publication (11). In essence, it is a mathematical representation of the anatomy, pathophysiology, signs, symptoms, behaviors, tests, treatments, logistics, resources, and outcomes associated with type 1 and type 2 diabetes, as well as several other diseases and conditions (CHD, congestive heart failure, asthma, stroke, hypertension, and obesity). The model uses an object-oriented approach and differential equations to recreate a level of detail corresponding to that in patient charts, medical textbooks, clinical practice guidelines, and clinical trials. It covers a spectrum of disease-related areas, taking into account biological details, care processes, patient and practitioner behaviors, logistics, resources, costs, and quality of life. The model represents biological variables and time continuously (any event can occur at any time) and, unlike Markov modeling, has no distinct states or strata. It is capable of modeling diseases and complications simultaneously, enabling it to address comorbidities, syndromes, multiple treatments, and treatments with multiple effects. Validation of the Archimedes Model against clinical trials has been published, providing details of 74 validation exercises involving 18 trials (12).

At the Fourth Mount Hood Challenge, the Archimedes Model presented the results of a publicly announced, blinded prospective prediction of the CARDS trial outcomes based on trial design information (e.g., inclusion/exclusion criteria, treatment protocols, and definitions of end points) and the prerandomization baseline patient characteristics, but without any postrandomization data (14). The CARDS investigators conducted a specific analysis of the outcomes estimated by the Archimedes Model (35). For type 1 diabetes, the Archimedes modelers presented the results of a previously published validation of the model against the primary end points of the DCCT trial: microalbuminuria, proteinuria, and retinopathy in the primary prevention and secondary intervention cohorts (12).

The first challenge for the Mount Hood 4 modelers was to simulate the progress of the cohorts in the CARDS trial over a time horizon of 4 years taking into account macrovascular complications by therapy allocation. The end points simulated by the models did not exactly accord with the reported CARDS end points (16) (Table 1), and further details can be found on the Mount Hood 4 Web site (13). For example, while most models attempted to simulate total MI (fatal plus nonfatal MI), the relevant published CARDS end point for comparison was “acute coronary events,” which also included hospitalized unstable angina, silent MI, and resuscitated cardiac arrest. Full definitions of end points for each model and those used in the CARDS study are available on the Mount Hood 4 Web site (13).

The 3.9-year total event rate of acute coronary events in CARDS was 5.5% in the control group and 3.6% in the intervention arm (10 mg atorvastatin daily). Most of the models taking part in the challenge reported total event rates of MI over a 4-year time horizon in the range of 5.3% (UKPDS Outcomes Model) to 8.0% (UKPDS Risk Engine) for the control group and 3.4% (Archimedes Model) to 5.7% (Sheffield Diabetes Model) for the intervention group (Table 1). (The Archimedes group presented 4.5-year cumulative incidence rates for acute coronary events at the meeting; 4-year results were also reported and are shown here for comparability with other models.)

Model estimates of absolute risks of stroke were close to those experienced in the trial by either the control or intervention group. However, the models did not generally predict the large reduction in risk experienced by the intervention group in the trial (Table 1). In the control group, CARDS reported a cumulative incidence of total stroke (fatal plus nonfatal) of 3.2% over a median 4-year follow-up period. This compared with model simulation values ranging between 1.7% (CDC/RTI Type 2 Diabetes Progression Model) and 3.9% (Sheffield Diabetes Model) for the participating models. In the atorvastatin intervention group, the cumulative incidence of stroke was 1.7% in CARDS, which compared with a range of values from 1.5% (CDC/RTI Type 2 Diabetes Progression Model) to 3.5% (Sheffield Diabetes Model) in the participating model simulations.

In terms of reproducing the CARDS results, most of the models appeared to slightly overestimate the incidence of macrovascular complications. One reason for this might be that the type 2 diabetic population in CARDS was a specially selected “low mortality risk” group, with no previously documented history of CVD or preexisting major illnesses. As a result, the incidence of macrovascular complications may be lower than in other type 2 diabetic populations such as the UKPDS. Previous studies have indicated that using the Framingham risk formula in diabetic populations underestimates the incidence of CVD (36–39). Many of the models participating in the meeting used the UKPDS Risk Engine to estimate the incidence of cardiovascular complications. (Exceptions were EAGLE, UKPDS Outcomes Model, and Archimedes.) The version of the Risk Engine available in 2004 is optimized for use in the early years of diabetes (3), whereas the CARDS population had a mean duration of diagnosed diabetes of almost 8 years. The Cardiff Diabetes Forecaster, which adjusts the Risk Engine for duration of diabetes, overestimated coronary risk to a lesser extent than the Risk Engine itself (Table 1).

Another possible reason for the apparent discrepancy is that when undertaking the CARDS challenge, each group had to make assumptions regarding the characteristics of the population used in the simulation. Groups used a variety of different methods that ranged from assuming patients had characteristics such as A1C equal to the reported mean values for the CARDS population and to the creation of synthetic populations using external data. We have provided summary statistics for the patient characteristics used by each group in their simulations on the Mount Hood 4 Web site. In some instances there were significant differences between the mean values for some characteristics and those reported for the CARDS population. For example, the Archimedes Model created a population in which 42% of the population was female, compared with only 32% in the CARDS study population. Other models, for example, the UKPDS Outcomes Model, did not directly incorporate any therapy-specific information, such as whether patients were on ACE inhibitors, on the assumption that their effect is captured via their impact on included risk factors such as systolic blood pressure. (See Mount Hood 4 Web site for summary statistics of the patient population used by each group in their simulations.) The influence of such differences on the reported results could not be fully resolved during the meeting but is an important area for further research, as are the methods used to construct these populations. This point also highlights the importance of clinical detail in diabetes models. Outcomes such as CVD are affected by a variety of risk factors. All of these risk factors need to be considered in a model to accurately simulate disease progression, and, of course, accurate input data are essential to producing realistic outcomes using any simulation model.

Three models at the Fourth Mount Hood Challenge were designed to specifically simulate type 1 diabetes: the EAGLE Model, the CORE Diabetes Model, and the Archimedes Model. In the cases of the EAGLE and CORE Models, this was achieved using a modular approach, with distinct approaches applied to the simulation of type 1 versus type 2 diabetes in keeping with the distinct etiologies of disease. The Archimedes Model includes both type 1 and type 2 diabetes in a single integrated model. All of the other models at Mount Hood 4 were designed only to simulate the progression of type 2 diabetes. The Archimedes group performed its validation against the DCCT 2 years ago and reported those previously calculated results at the Mount Hood 4 meeting.

For the simulation of nephropathy complications, the DCCT reported a cumulative incidence of microalbuminuria (urinary albumin excretion rate 30–300 mg/24 h) of 27.3% in the conventional treatment arm compared with 16.0% in the intensive treatment arm for the primary prevention cohort. These values were projected to be 27.6 and 14.9% by the CORE Diabetes Model and 28 and 15% by the Archimedes Model. The EAGLE Model projected higher cumulative incidence rates due to the fact that the secondary prevention cohort was used as basis for simulations (41.7 and 25.3% in the conventional and intensive treatment arms, respectively). The DCCT reported 9-year cumulative incidences of microalbuminuria of 42.1% in the conventional treatment arm and 26.2% in the intensive treatment arm in the secondary prevention cohort. As all groups had used the DCCT to develop their model, these should be regarded as internal validation exercises (1).

The development of background retinopathy, defined as a three-step change in patients without retinopathy at baseline using the retinopathy assessment scale for the DCCT, was also projected (Table 2). The CORE Diabetes Model projection of cumulative incidence of diabetic retinopathy in the intensive treatment group corresponded closely to the DCCT data (14.4 vs. 14.3%, respectively), but less so in the conventional treatment group (39.4 vs. 52.2%, respectively). The Archimedes Model projected comparable values of 13 and 40% for the two groups, respectively. The EAGLE group, which used data from the WESDR to build its model of retinopathy (proliferative and nonproliferative), calculated higher incidence rates for retinopathy than reported by the DCCT, providing values of 69.6 and 76.9% in the intensive and conventional treatment groups, respectively. It is worth noting that for this end point, the 9-year value from the DCCT had a wide range of uncertainty due to the very small number of patients in the study for that duration (<50 people).

A similar pattern was observed for the onset of peripheral neuropathy end point. The 9-year cumulative incidence of neuropathy values (defined as abnormal nerve conduction) extrapolated from DCCT data for the primary prevention cohort were 63.2 and 27.7% for conventional therapy and intensive therapy, respectively (22). Projections using the CORE Diabetes Model produced values of 64.0 and 25.0%, respectively, for peripheral neuropathy. The reported cumulative incidence after 5 years was 40.2 and 16.5% for the conventional and intensive arms, respectively, in the DCCT. The EAGLE Model used the definition of peripheral neuropathy identified by clinical examination and confirmed by subsequent investigation (21.3 and 10.0% in the conventional and intensive groups of the extrapolated DCCT data, respectively) and projected cumulative incidences of 18.1 and 5.2% for the conventional and intensive groups, respectively (derived from the weighted mean of the intensive group). The reported cumulative incidences of peripheral neuropathy after 5 years were 15.2 and 6.9% for the conventional and intensive arm, respectively, in the DCCT. For neuropathy, the Archimedes Model calculates the Semmes-Weinstein 20 g test (40), which was not reported by the DCCT.

In a methodological session at the Mount Hood Challenge 4 Meeting, modelers were invited to summarize their validation results to date. Several groups reported “external validations”— tests against data independent from the model. The Cardiff group reported that their model had successfully predicted cardiovascular event rates in the Cardiff Diabetes Registry. Since the model derives cardiovascular risk from the UKPDS Risk Engine, not Cardiff data, this is an external validation. The UKPDS group reported a validation of the current Risk Engine by an independent group in a community-based cohort with newly diagnosed diabetes. The CHD event rate predicted by the Risk Engine was close to the observed rate (within the 95% confidence interval) (37). The Cardiff result mentioned above also supports use of the current Risk Engine. The CORE group presented a mixture of external and internal validation results against many studies, with the external validations predominantly successful (8). The Archimedes group presented previously published results for of external validations against nine clinical trials, of which eight were completely successful and the ninth partially successful (12). Of the other models, for EAGLE and the UKPDS Outcomes Model results were reported from successful internal validations: tests against the data on which the model was built. The ADA guidelines for diabetes modeling encourage validation, preferably external validation, wherever possible (1). The DiDACT group argued that validation has limited value because validation exercises (e.g., comparison to 4-year results from CARDS) are narrow compared with real applications (e.g., estimating lifetime outcomes). Others argued that a limited test of a model is preferable to no test.

The second methodological session was used by different speakers to emphasize different aspects of statistical uncertainty. The UKPDS group presented results from their Outcomes Model that demonstrated the importance of eliminating first-order uncertainty, which is the uncertainty that arises when Monte Carlo simulation is the method used for model calculations. Some models (CDC, UKPDS Risk Engine, and DiDACT) use calculation methods that are not subject to first-order uncertainty. Others (CORE, Cardiff, EAGLE, Sheffield Diabetes Model, UKPDS Outcomes Model, and Global Diabetes Model) eliminate first-order uncertainty by averaging over multiple simulation runs. Results from Archimedes are based on single simulation runs. “Second-order” uncertainty, statistical uncertainty in model parameters, is addressed via confidence intervals in some models and via probabilistic sensitivity analysis in others (1). No single group presented results on model uncertainty or variation in results due to different design decisions in modeling (41). The Mount Hood Meetings can be seen as an investigation of this important source of uncertainty that is often overlooked because it is difficult to quantify in any single model (1).

An important discussion point at the Fourth Mount Hood Challenge was the issue of the meeting's limitations and whether it offers a fair comparison across all diabetes models. The DiDACT Model group declined to compete in Mount Hood Challenge 4, arguing that none of the end points presented in the challenge were comparable with their model outputs (as in most cases they did not use the same measures). It was emphasized that, although discrete event simulation models are most widely used, other types of model exist that have a serious and significant contribution to make to diabetes modeling.

In past Mount Hood Challenge meetings, participants have compared outcomes for hypothetical diabetes cohorts and interventions. While interesting comparisons were made and important differences between models identified, the format of previous Mount Hood Challenges was limited in that there were no “real-life” data to use as a yardstick. As a result, it was not possible to test the external validity of the participating models' forecasts. Previous Mount Hood Challenges were restricted by incomplete cohort and treatment definitions; therefore, assumptions on key input parameters had to be made. Moreover, each modeling group applied its own utility functions and cost settings, leading to different assumptions being made by each group, which contributed to the divergence in forecast outcomes.

Mount Hood Challenge 4 was designed with these previous limitations in mind and taking into consideration the fact that key steps toward having diabetes models widely accepted by decision-makers and physicians are those of transparency and validity. Models must be understood by their target audience and trusted by the end users. It therefore seems essential that modelers make every effort to clearly describe and validate their models and quantify and explain predictive differences between models. Such issues are not confined to diabetes modeling; a recent exercise in which researchers developed seven independent statistical models of breast cancer incidence and mortality to assess the effects of screening and adjuvant therapy reported substantial variability across models (42). With this drive toward validity for end users and the growing demand for accurate diabetes models, especially for a disease with such a huge financial burden in a global climate of limited health care resources, meetings such as the Mount Hood Challenge, where competing modeling groups compare and contrast models with clinical trial data, could have an increasingly important role to play in coming years.

Andrew J. Palmer, MBBS (CORE, Basel, Switzerland); Stéphane Roze, MSc (CORE, Basel, Switzerland); William J. Valentine, PhD (CORE, Basel, Switzerland); Philip McEwan, PhD (University of Cardiff, Cardiff, U.K.); Michael Gillett, BSc, CIMA (University of Sheffield, Sheffield, U.K.); Michael Holmes, BSc (University of Sheffield, Sheffield, U.K.); Philip Clarke, PhD (UKPDS, University of Oxford, Oxford, U.K.); Richard Stevens, PhD (UKPDS, University of Oxford, Oxford, U.K.); Alastair M. Gray, PhD (UKPDS, University of Oxford, Oxford, U.K.); Ruth Coleman, MSc (UKPDS, University of Oxford, Oxford, U.K); Stephen Sorensen, PhD (Centers for Disease Control and Prevention, Atlanta, GA); Elvira Müller, PhD, MPH (EAGLE Model, Analytica International, Lörrach, Germany); Stefan Walzer, MA (EAGLE Model, Analytica International, Lörrach, Germany); David M. Eddy, MD (Archimedes Model, Kaiser Permanante, San Francisco, CA); Richard Kahn, PhD (ADA, Alexandria, VA); Adrian Bagust, PhD (DiDACT Model, University of Liverpool, Liverpool, U.K.); and Jonathan Brown, PhD (Global Diabetes Model, Kaiser Permanente, Northwest Region, Portland, OR).

Alan Brennan, MSc (University of Sheffield, Sheffield, U.K.); Wiley Chan, MD (Global Diabetes Model, Kaiser Permanente, Northwest Region, Portland, OR); Alan Russell, PhD (Kaiser Permanente, Southern CA); Thomas Hoerger, PhD (Centers for Disease Control and Prevention, Atlanta, GA); Katherine Hicks, MSc (Centers for Disease Control and Prevention, Atlanta, GA); Roman Casciano, MSc (EAGLE Model, Analytica International, Lörrach, Germany); Rito Bergemann, MD (EAGLE Model, Analytica International, Lörrach, Germany).

The Fourth Mount Hood Challenge Meeting was supported by an unrestricted grant from Novo Nordisk and hosted by the Center for Outcomes Research in Basel, Switzerland. Richard Stevens is funded by the Health Foundation.

The Mount Hood 4 modelers thank Novo Nordisk A/S for the provision of an unrestricted grant to support the Fourth Mount Hood Challenge and Professor Helen Colhoun and the CARDS group for their support in preparing this article. We would also like to thank the Center for Outcomes Research and in particular Josh Ray for their work behind the scenes to ensure the smooth running of the meeting.