Simulated Physician Learning Intervention to Improve Safety and Quality of Diabetes Care: A Randomized Trial

A group randomized trial was designed to test the hypothesis that primary care physician (PCP) participation in either of two study interventions would improve safety or quality of care delivered to actual patients with diabetes, relative to care delivered by control group PCPs. Four principal dependent variables were measured in actual patients: 1) change in A1C and LDL cholesterol testing rates, 2) change in A1C and LDL cholesterol levels, 3) rates of intensification of glucose or lipid medication when patients are not achieving recommended clinical goals, and 4) change in risky prescribing events related to glycemic control.

The study was conducted at HealthPartners Medical Group, an 18-clinic multispecialty group that provides care to 8,000 adults with diabetes. At the end of 2001, the mean A1C in the 90% of diabetic patients who had an A1C test within a year was 7.2%, and the mean LDL cholesterol was 110 mg/dl, reflecting relatively good baseline diabetes care (11). Approximately 10% of adults with diabetes consulted an endocrinologist each year, most often for one visit (18). Additional information about the study site and antecedent efforts to improve diabetes care can be found in previous publications (11).

A sample of 122 potentially eligible PCPs was identified. Physicians in the sample were employed at least 20 h a week at HealthPartners Medical Group as of 1 January 2000 and provided care to a minimum of 10 adult diabetic patients in 2000. Participating PCPs received compensation of $100 for group A, $200 for group B, or $600 for group C, predicated on the differential time commitment to each intervention. Before randomization, 67 consenting physicians were blocked into groups of three based on 1) same specialty (family medicine or internal medicine) and 2) whether they provided care to <50 vs. 50 or more diabetic patients. The intervention was done in early 2002, and patient data to assess impact of the intervention included data from 1 January 2001 to 31 December 2003.

Diabetic patients who received care from study physicians were selected for analysis if they met these criteria: 1) were enrolled in HealthPartners Medical Group on 1 January 2000, 2) were at least 19 years of age on 1 January 2000, or 3) had diagnosed diabetes in 2000 based on meeting at least one of these criteria: two or more outpatient ICD-9 diagnostic codes 250.xx for diabetes or filled one or more prescriptions for a diabetes-specific drug. This diabetes identification method has estimated sensitivity 0.91, specificity 0.99, and positive predictive value 0.94 (19). To be included in the analysis, eligible patients had to have the same study-participating PCP for 2 years before and 1 year after the intervention. Analysis included 2,020 eligible adult diabetic patients.

PCPs randomized to this group completed baseline surveys but received no intervention.

Using software developed in previous work (17,20), we provided each PCP in group B with three clinical scenarios in the same fixed order. An electronic medical record–like interface permitted multiple virtual patient-physician encounters with each case in the presence of a research assistant with no clinical training. At each simulated patient encounter, the PCP viewed history and physical exam data, recorded impressions, and took a series of actions that were not scripted and could include ordering tests, making referrals to specialists and educators, recommending diet and physical activity, and initiating or titrating various medications for glucose, blood pressure, lipids, or depression.

Actions could be taken at each scheduled visit or phone contact. Follow-up was scheduled at any interval recommended by the physician. At the next encounter, the patient's clinical data (A1C, LDL cholesterol, and blood pressure levels and other data) reflected the effect of actions taken in previous encounters, attenuated by the time-dependent effects of medications, lifestyle recommendations, and recognition and treatment of depression. At each follow-up encounter with the simulated patient, the physician received “learning by doing” feedback in the form of patient responses to actions taken in previous encounters. Each physician dealt with three simulated cases over 60 min; each case was treated for a series of simulated encounters over variable lengths of simulated calendar time. At the end of the three cases, each physician received a printed feedback record of the actions they had taken in each case compared with actions taken by an expert physician who performed the same cases. A more complete description of this intervention software is available (17).

The patient cases seen by each physician were initialized for three important clinical situations: 1) a newly diagnosed type 2 diabetic patient on no medications, 2) a patient with contraindications to insulin sensitizers (metformin and thiazolidinediones) who required insulin initiation and subsequent titration, and 3) a depressed individual with resulting low adherence who required insulin titration.

Physicians in group C received the same three simulated cases as group B, the same “learning by doing” feedback based on actions taken, and the same printed feedback summary of actions taken compared with those of an expert physician. In addition, upon completion of the three cases, group C physicians received 60 min of verbal interaction and feedback from a physician opinion leader who observed the physician while he/she performed the simulations and used a predesigned checklist at the completion of the three cases as a tool to discuss potentially problematic treatment issues, as well as to give positive reinforcement for good practice patterns that were observed. Therefore, opinion leader feedback included both positive and negative aspects of physician performance.

A1C and LDL cholesterol test dates were obtained from electronic clinical databases and were used to assess physician adherence to recommended frequency of these tests in adults with diabetes.

Glycemic and lipid control were assessed by comparing the last preintervention A1C and LDL cholesterol with the last A1C and LDL cholesterol value in the 12-month postintervention period. All A1C assays were performed for routine care at a single, centralized, accredited clinical chemistry laboratory. No changes in assay procedures occurred during the study. Details of the A1C assay (21) (normal range 4.5–6.1%; coefficient of variation 0.058% at A1C 8.8%) and LDL cholesterol assay (22) (calculated only after a 12-h fast and for triglycerides <400 mg/dl) are described elsewhere (11).

Medication moves (including initiation or titration of glucose or lipid drugs) were determined in the 12-month postintervention period for all diabetic patients with a baseline A1C ≥7% or LDL cholesterol ≥100 mg/dl. Medication moves were identified based on drug doses and prescription fills during defined periods of time using pharmacy claims data. Because insulin is often discarded monthly due to expiration, insulin claims data do not accurately reflect actual usage; therefore, insulin dose titration was excluded from the glucose move variable.

The learning intervention focused on one risky prescribing event: use of metformin in patients with serum creatinine values ≥1.5 mg/dl. By design, the learning intervention did not address risky use of metformin in patients with congestive heart failure (CHF).

Analysis was limited to two levels (patients nested within physicians) because intraclass correlation coefficients at the clinic level were small, consistent with previously reported data. Patient-level independent variables obtained from electronic databases included age, sex, and baseline comorbidity score (23). Physician-level independent variables obtained from electronic databases included age, sex, specialty, years of experience, and number of diabetic patients.

This group randomized trial was analyzed as a nested (patients nested within physicians) cohort pretest-posttest control group design when the variables being examined were available both pre- and postintervention (e.g., A1C value). In this analysis, the coefficient and P value for the time-by-condition interaction term tested the intervention effect by examining differences in condition (groups A, B, and C) and change over time. In some instances, post hoc contrasts were utilized to compare change over time in specific combinations of conditions (24). Nested analysis of postintervention data with regression adjustment for patient age, sex, and comorbidity score was used when examining variables measured only postintervention (e.g., medication moves). In this type of analysis, the intervention effect was tested by the coefficient and P value for condition. This analytic approach was utilized as an alternative method to the time-by-condition analysis in order to assess the effect of study arm after control for patient covariates. Since results using this method were similar to those obtained in the time-by-condition analysis, we report only the time-by-condition results.

Mixed-effects models were estimated with SAS Proc Mixed. For dichotomous outcomes (e.g., test rates), Proc Mixed was also used with and without the Glimmix macro. Because results were similar with and without this macro, we report the more readily interpretable (24) results from models without the Glimmix macro.

A priori sample size calculations assumed 20 providers with 30 diabetic patients each (600 diabetic patients per study arm) would be available for analysis. Effective patient sample size was estimated as n = 311 per arm due to clustering using a measured intraclass correlation coefficient = 0.032 based on eligible physicians. This study was designed with 80% power to detect an A1C difference of 0.3% between study arms, with a two-tailed α = 0.05. This study was reviewed, approved in advance, and monitored by the HealthPartners Institutional Review Board, project no. 00-016. Informed consent was obtained from all physician study subjects. This study is registered as no. NCT00262704 at www.clinicaltrials.org.

We were able to recruit >50% of eligible PCPs to participate in the study. Physician attrition occurred due to noncompletion of the physician baseline survey (n = 1), withdrawal from the study (n = 1), noncompletion of the intervention (n = 3), and departure from the medical group during the study period (n = 3). Two physicians were excluded from analysis because they had insufficient postintervention diabetic patients available due to changes in job responsibilities. Attrition occurred evenly across randomized groups, and final analysis included 19 physicians in each group. Over 97% of those who completed the learning intervention rated their satisfaction with the interventions as excellent or very good after completing the simulated cases.

Table 1 characterizes patients and physicians in each of the three study arms. Randomization at the physician level resulted in similar patient samples except that patients of physicians in group B more often had coronary artery disease and higher Charlson scores. Physician attributes did not differ by group.

Table 2 shows changes in A1C testing and A1C levels. There was a significant change in A1C level from before to after intervention, and the time-by-group interaction term was significant (P = 0.04), indicating significant group differences in change in A1C levels over time. A1C values of patients of physicians in group B declined, while the A1C values of patients of physicians in groups A (control) and C increased. An analogous model (data not shown) predicting postintervention A1C levels from each group and controlling for preintervention A1C, patient age, sex, and preintervention Charlson score showed similar results.

Table 2 also shows results of the analysis restricted to patients on insulin for ≥120 days in both the pre- and postintervention periods. The overall time-by-condition term indicates no significant postintervention group differences in A1C change in insulin-treated patients. Patients not on insulin had an overall A1C increase of 0.13% from pre- to postintervention, with no differences by group (P = 0.10). Most of the A1C advantage of group B patients was concentrated in the subgroup using insulin.

Table 2 also shows that LDL cholesterol values improved over time (P < 0.001), with no significant differences in LDL cholesterol test rates (P = 0.30) or LDL cholesterol values (P = 0.67) across study groups. Statin use (P < 0.0001) and fibrate use (P = 0.001) increased over time with no differences across study groups (time-by-group P = 0.22 for statins and P = 0.78 for fibrates).

Among patients above A1C goal during the preintervention time period (n = 907), the rate of postintervention initiation or titration of blood glucose–lowering medications was 31.5% in group A, 32.6% in group B, and 36.8% in group C (P = 0.41). Insulin titration could not be ascertained from data sources. Among patients with their most recent preintervention LDL cholesterol ≥100 mg/dl (n = 701), initiation and titration of lipid medications was 20.8% in group A, 24.2% in group B, and 21.8% in group C (P = 0.66). Among patients with their most recent preintervention blood pressure >130/80 mmHg (n = 949), the rate of postintervention initiation or titration of blood pressure–lowering medications was 27.3% in group A, 24.7% in group B, and 28.2% in group C (P = 0.61). Thus, the intervention did not significantly increase rates of treatment intensification.

Table 3 provides data on risky prescribing events. While the time-by-condition term was only marginally significant (P = 0.08), a contrast of the control condition (group A) with the combined intervention groups (B plus C) showed a significant effect (P = 0.03) in favor of groups B and C for reduction in metformin use in patients with a renal contraindication to metformin use. No changes by group in metformin use following a CHF diagnosis (P = 0.36) were observed, consistent with the fact that this issue was not highlighted in the clinical cases.

Despite the brevity of the learning interventions, we observed significant effects on diabetes care delivered to actual patients in the 12 months after each intervention was completed. First, both intervention groups showed an ∼60% decline in risky metformin use in patients with reduced renal function compared with the control group. Notably, there was no change in risky use of metformin in CHF patients, which was a clinical issue not included in the learning cases. Second, patients of physicians in group B had more desirable patterns of glycemic control postintervention than patients in the other groups. Finally, while lipid control improved in all groups, no differences in LDL cholesterol levels were observed across intervention groups.

Based on postintervention changes in treatment patterns in real patients, we conclude that different things were learned by the physicians in our two intervention groups (B and C). The part of each intervention that was common to both physician groups was comprised of three simulated cases, two of which focused on insulin initiation and titration. The data suggest, but do not prove, that physicians in group B learned to make additional insulin titration moves for their patients who were already on insulin (see Table 2), and better management of insulin-treated patients led to the A1C improvement we observed in group B.

The physician opinion leader did not enhance the simulated learning intervention. The reason for this is unclear but could have included the following: 1) the opinion leader focused on less-relevant clinical issues than the simulated program itself, 2) the presence of another physician observing increased anxiety of the physician learner, or 3) the positive reinforcement of the opinion leader led the providers to believe they did not need to change their clinical behaviors with actual patients. This finding is encouraging from a cost perspective because opinion leader time adds significant cost to the intervention.

There are a number of factors that limit the interpretation of these data. First, the study was conducted among PCPs at a single medical group, and generalization of results to other settings is not assured. Second, the lack of data on insulin intensification and patient medication adherence constrains our analysis of how interventions mediated their effects. Third, because blood pressure values were not yet electronically available at the time of data analysis, we were unable to assess the impact of the intervention on changes on blood pressure, although the learning cases included blood pressure management issues. Fourth, the study was conducted at a site with relatively good baseline diabetes care, and intervention impact may differ in settings where baseline care is poorer. Finally, the study design did not assess whether augmentation of the interventions using additional cases or periodic repetition could further amplify the benefits of this limited intervention involving only three diabetic case subjects.

Despite some limitations, study results are interesting and important. Safety improvement was substantial, and although glycemic control improvements were modest, few previous physician-learning interventions have reported positive results (25). This learning technology could be more effective if simulated cases were customized for each individual physician based on analysis of patterns of care in electronic health records. Further experimentation with this intervention strategy in other settings and for other clinical conditions is also warranted. Effective physician learning interventions such as these, which are brief, enjoyable, and scalable, may complement other care improvement strategies and may contribute to the essential goal of improving the safety and quality of chronic disease care.

This project was supported by the Agency for Healthcare Research and Quality (grant no. RO1 HS 10639).

No potential conflicts of interest relevant to this article were reported.

All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.