Estimating Physician Effects on Glycemic Control in the Treatment of Diabetes: Methods, effects sizes, and implications for treatment policy

We created a cohort of patients with type 2 diabetes at a Veterans Affairs medical center in the southeastern U.S. using multiple patient files and existing Veterans Health Administration data sources. Individuals with type 2 diabetes were identified based on having at least two ICD-9 codes for diabetes and having two or more visits each year since their diagnosis based on a previously validated algorithm (17). The cohort included 14,931 patients. For this study, we selected patients from the cohort who had at least two A1C measurements within 1 year apart and who had a specific designated primary care provider within the same period. To ensure reliability of study estimates, we excluded patients whose providers had less than five diabetic patients. The resulting dataset consisted of 1,381 patients nested within 42 physicians. On average, each study physician had (mean ± SD) 33 ± 12.99 patients with diabetes.

Patients in the study sample were 98% male. The mean age ± SD of patients in the sample was 66 ± 11 years and the baseline A1C was 7.52 ± 1.97. Race and ethnicity measures were aggregated to create three racial/ethnic groups: non-Hispanic whites (whites, 52%), non-Hispanic blacks (blacks, 38%), and Hispanics, Asians, multi-racial, and unknown (other, 10%).

Preliminary HLM analyses were used to investigate the distributions of patient characteristics among study physicians to assure random distribution and identify potential covariates. There were no statistically significant differences among physicians in regards to patient sex, minority status, systolic blood pressure, diastolic blood pressure, or weight. Statistically significant, though small, differences were found in baseline A1C (<1% of the variance) and percentage of patients aged 65 years (3% of the variance). Thus, both were identified as covariates in the full model.

The outcome measure of interest in the current study was change in A1C from baseline. A1C measures the average percentage of blood glucose over the prior 2–3 months and is considered an index of how well glucose is being controlled. Complications from diabetes can be prevented or delayed if A1C levels are kept at <7% (18). For the current study, A1C was measured at baseline and again at an average of 12 months after baseline (mean ± SD 363 ± 36 days).

HLM was used to address the current research questions because it allows variance in outcomes to be partitioned between patients and physicians. This is possible due to the method of handling of error. In regular multiple regression models, prediction error associated with patient characteristics and prediction error associated with physician specification would be combined into one error term. In contrast, HLM allows each physician to have a unique intercept (and slope) for patient effects on outcomes. Thus, error attributed to patient effects and error attributed to physician effects are in different terms (19). Statistical analyses in the current study were performed using HLM6 software (20). Veterans Affairs data are well suited for such an analytic strategy because patients are randomly assigned to available physicians. This limits confounding factors by naturally encouraging (though not guaranteeing) an orthogonal relationship between patient and physician characteristics.

To investigate the role of physician effects on A1C change over and above patient-level factors, a hierarchical unconditional model was used. In HLM, such models are called unconditional because they do not include specific predictors. Their function is to partition variance in outcomes among level 1 and level 2 unit specifications (21). An unconditional model was used to test the null hypothesis that there was no variation in A1C due to physician-level factors. To do this, variance components σ² and τ were estimated for patient-level and physician-level error terms, respectively. The null hypothesis could be safely rejected if variance component τ was significantly different from zero and significantly improved model fit. Significance for model fit was based on comparing pairs of model deviances distributed asymptotically on a χ² distribution (20).

Given significant results, unconditional model effect sizes were estimated using intraclass correlations (ICCs) consistent with the methods recommended by Bryk and Raudenbush (21). In a two-level hierarchical structure, the ICC is defined as the proportion of the total variance in the outcome variable that is between the level 2 units, or R²_total (19). In a hierarchical patient-physician model, the ICC represents the proportion of variance accounted for at the physician level. It therefore follows that the remainder of the variance can be attributed to factors at the patient level or to other unspecified nested factors.

To investigate if and how physician effects varied across patient populations, a simple effects design was used. Separate unconditional models were estimated for blacks, whites, others, patients aged 65 years, patients aged <65 years, and patients who improved, stayed the same, or got worse over the course of the treatment year. Patient improvement was defined as a ≥15% decrease in A1C from baseline (in the study sample, this averaged to a 1-point overall decrease in A1C). Patient worsening was defined as a ≥15% increase in A1C, and patients who stayed the same were defined as those whose A1C change was <15% in either direction.

To investigate the role of available patient characteristics in diabetes outcomes, a stepwise random coefficients model was estimated. The model was constructed by adding specific patient-level predictors to the total-sample unconditional model. Predictors were entered in the following order: baseline A1C, age, and race/ethnicity (blacks, whites, or others), with each race/ethnic distinction dummy coded and alternately added for effect estimation. With the current sample size (n = 1,381), it is assumed that coefficient estimates for patient age and race are reliable and valid for similar populations; however, mathematical coupling and regression to the mean pose potential threats to the precision of the correlation between baseline A1C and change in A1C (22). Accordingly, baseline A1C is included as a peripheral control rather than as a definitive measure of the impact of initial disease on A1C change.

As mentioned, within an HLM framework the ICC or R²_total indicates the amount of total variance accounted for by level 1 and level 2 factors, regardless of what those factors are. By adding specific predictors to each level, it becomes possible to estimate R²_within and R²_between. In the present study, R²_within was estimated for each level 1 predictor in a manner consistent with the recommendations of Snijders and Bosker (22) and denoted what portion of the total variance attributed to patients could be accounted for by each patient predictor. R²_between was not estimated because no specific physician-level factors or characteristics were entered into the models.

The mean ± SD A1C after a year in treatment was 7.21 ± 1.68. Table 1 shows the estimated variance components (σ² = 5.14; τ = 0.09) for patient- and physician-level units in the overall-sample unconditional model. The value of τ was significantly different from zero and improved model fit (χ² = 71.77; d.f. = 41; P = 0.002), indicating the presence of physician-level effects beyond patient-level factors. The ICC was 0.02, indicating that 2% of the overall variance in A1C change over the study period could be accounted for by physician-level factors and 98% could be attributed to patient-level or unspecified factors. As Table 2 indicates, adding patient covariates of age, baseline A1C, and nonminority status to the unconditional model had no substantial effect on the ratio of variance accounted for by physician specification (ΔICC = 0.005). Figure 1 shows the amount of variance in A1C change accounted for by patient and physician factors.

The unconditional model for black patients yielded a physician-level variance component (τ = 0.18) that was significantly different from zero and improved model fit (χ² = 66.22; d.f. = 41; P = 0.02), indicating the presence of physician-level effects beyond patient-level factors. The ICC indicated that 3% of the overall variance in A1C change for blacks could be accounted for by physician-level factors. The unconditional model for whites yielded a physician-level variance component (τ = 0.08) that was not significantly different from zero and did not improve model fit (χ² = 55.52; d.f. = 41; P = 0.06). However, marginally significant findings for whites may have been due to a limited sample size rather than to a true difference between groups. The nonsignificant ICC indicated that 2% of the overall variance in A1C change for whites could be accounted for by physician-level factors.

The unconditional model for patients aged 65 years yielded a physician-level variance component (τ = 0.13) that was significantly different from zero and improved model fit (χ² = 60.64; d.f. = 41; P = 0.02). The ICC was 0.02, indicating that 2% of the variance in A1C change for patients aged 65 years could be accounted for by physician-level factors. The unconditional model for patients aged <65 years yielded a physician-level variance component τ (= 0.01) that was not significantly different from zero (χ² = 47.54; d.f. = 41; P = 0.22).

The unconditional model for patients who improved yielded a physician-level variance component (τ = 0.14) that was significantly different from zero and improved model fit (χ² = 68.20; d.f. = 41; P = 0.005). The ICC was 0.05, indicating that 5% of the overall variance in A1C for patients who improved could be accounted for by physician-level factors. The unconditional model for patients who did not improve yielded a physician-level variance component (τ = 0.01) that was not significantly different from zero and hence did not improve model fit (χ² = 48.37; d.f. = 41; P = 0.17). The unconditional model for patients who got worse yielded a physician-level variance component (τ = 0.00) that was not significantly different from zero and hence did not improve model fit (χ² = 28.25; d.f. = 40; P > 0.50).

The random coefficients model predicting A1C change estimated effects for patient A1C baseline, patient age, and patient race. The first factor entered into the model was baseline A1C. As Table 2 indicates, baseline measurement significantly predicted A1C improvement over the course of the treatment year (t = −31.94; d.f. = 1,380; P = 0.000). Higher baselines were associated with greater decreases in A1C. The estimated R²_within was 0.46, indicating that baseline levels accounted for 46% of the patient-level variance in A1C change or 45% of the total variance (46 of 98%).

Patient age significantly predicted A1C change over 12 months (t = −2.27; d.f. = 1,380; P = 0.023) after controlling for baseline A1C. Older age was associated with greater A1C improvement. The estimated R²_within was 0.004, indicating that patient age accounted for 1% of patient-level variance in A1C change. Even so, the effect translated into measurable differences in health outcomes. A production function of the modeled age coefficient indicated that for a patient aged 55 years, age was associated with an improvement of 0.55 in A1C levels, whereas for a patient aged 75 years, the improvement was 0.75. This is an effect size of about one-tenth of an SD difference (d = 0.09).

The three racial/ethnic groups were alternately entered into the model. Controlling for baseline A1C and age, categorization as black was not associated with significant change in A1C (t = 0.15; d.f. = 1,380; P = 0.11) and neither was categorization as other (t = 0.04; d.f. = 1,380; P = 0.70). However, categorization as white was associated with significant improvement in A1C over the course of the treatment year (t = −2.33; d.f. = 1,380; P = 0.02). Because the predictor variable was dummy coded, indicating being white versus black or other, the effect is referred to here as nonminority status. The estimated R²_within was 0.003, indicating that nonminority status accounted for <1% of the patient-level variance in A1C change. The effect translated into a between-group difference of about one-tenth of an SD (d = 0.09). Although in general black patients showed greater improvement than white patients, that difference appeared to be due to black patients having higher baseline A1C. After baseline A1C was entered into the model, white patients showed greater improvement.

The primary goal of this study was to investigate whether diabetes outcomes varied by randomly assigned physicians over the course of a year in treatment. Physician-level factors were associated with statistically significant but modest variability in health outcomes, accounting for 2% of the total variance in A1C change. Simple effects analyses revealed that physician factors mattered more for black than for white patients and more for patients aged >65 years than for those aged <65 years. Interestingly, physician factors mattered most for patients who improved over the year, accounting for 5% of the total variance in A1C change. Physician-level factors did not matter at all for patients who got worse. Significant patient-level predictors of A1C change were baseline A1C, age, and nonminority status. The results of this study suggest that physician effects only have minimal impact on glycemic control. These findings are consistent with the results of a prior study by Krein and colleagues (16). The researchers investigated 12,110 patients within 258 primary care physicians across 13 Veterans Affairs medical centers and found that physician effects accounted for 1% of total variance in A1C. The current study builds on those findings by examining change over time in A1C and by investigating how physician effects vary by patient characteristics.

The significance and meaning of the physician-level effect reported here (2% of the variance) depend on one's perspective. Interpreting small effect sizes in large populations is often difficult. For example, in the U.S. population, Down's syndrome accounts for a statistically significant, though very small, amount of variance in IQ scores; however, one cannot assume Down's syndrome is not an important factor in IQ. Likewise, although physician effects appear to play a relatively small role in overall A1C outcomes for some individual patients or groups of patients, physician effects may be particularly important for survival and quality of life.

Alternatively, from a policy-oriented perspective, 2% of the variance does not seem a particularly fruitful target on which to focus limited resources and expensive interventions. Ninety-eight percent of the variance in A1C change in the current sample appears to be due to patient-level factors. Much of that variance is accounted for by largely unmutable factors such as baseline disease, age, and race. However, a good deal of the variance (52%) remains to be explored and may very well be associated with potentially mutable factors.

The results of this study suggest it is logical to support policies encouraging the development of patient-level interventions because that level accounts for the majority of variance in glycemic control. Moreover, significant progress has been made in the development of effective patient-oriented interventions. A systematic review of 72 randomized controlled trials (RCTs) in patients with type 2 diabetes showed that self-management training improved diabetes knowledge, frequency and accuracy of glucose self-monitoring, and glycemic control (24). A meta-analysis of 31 RCTs found that self-management education decreased A1C on average by 0.76% (95% CI 0.34–1.18), and the amount of decrease was highly correlated to the number of contact hours (25). Moreover, a recent meta-analysis of 25 RCTs investigating the effects of psychological interventions on A1C outcomes indicated improved outcomes for the experimental conditions in about one-half of the studies (26). It should be noted that these effects were in addition to physician care. In essence, studies that found significant results did so by manipulating some factor within the “Unspecified Patient Effects” portion of Fig. 1. Given the current state of medical treatments, that portion is where the largest potential payoff for future interventions exists.

Results of the current study are particularly relevant in light of the recent movement toward patient empowerment in diabetes self-care (27–28). As noted, results indicated that physician factors mattered the most for patients who improved and not at all for patients who got worse. The patient-empowerment literature could offer one straightforward explanation for this differential effect—patients who feel empowered may engage with their doctors more overall and, consequently, their doctors' skills, talents, and characteristics have more of an opportunity to affect outcomes. In contrast, patients who do not feel empowered to manage their diabetes may be less engaged with their physicians and thus the skills and characteristics of their physicians may not influence treatment outcomes. In such cases, “bad” treatment does not make patients worse and “good” treatment does not make them better. However, for patients who improved, physician-related factors made a meaningful difference in how much or how little they improved.

This study has some limitations. First, it is important to note that physician-level factors are not synonymous with physician characteristics. Potential nonrandom factors associated with physician specification, such as nursing team allocation or the availability of facility resources, may have the potential to bias estimates of physician-specific effects. Future investigations should take into account the ecology of hospital settings and sample from a variety of institutions. Second, there are important patient-level factors that were not available for this analysis. There is a need to further explore patient-level factors affecting diabetes outcomes, and future studies would benefit from including a broader range of patient-level predictors.

In conclusion, the results of this study suggest that it is logical to support policies encouraging the development of patient-level interventions because that level accounts for the majority of variance in glycemic control. Overall, results indicate a need for creativity in current treatment paradigms and policies, including the expanded use of multidisciplinary provider teams and behavioral approaches in the treatment of type 2 diabetes.

We thank Anouk Grubaugh, PhD, and Ron Acierno, PhD, Medical University of South Carolina, and Sim Galazka, MD, University of Virginia School of Medicine, for helpful reviews of the manuscript.