Is the Number of Documented Diabetes Process-of-Care Indicators Associated With Cardiometabolic Risk Factor Levels, Patient Satisfaction, or Self-Rated Quality of Diabetes Care? : The Translating Research into Action for Diabetes (TRIAD) study

The Translating Research into Action for Diabetes (TRIAD) study is a multicenter prospective study to identify successful strategies and modifiable barriers to optimal diabetes care. The design of TRIAD has been previously described (18). TRIAD enrolled patients with diabetes from six research centers collaborating with 10 health plans that serve ∼180,000 adults with diabetes in six U.S. regions. Eight of the health plans contracted, with a total of 68 provider groups, to deliver care to enrolled patients. Three plans covered some or all patients through contracts with individual physicians, without provider group involvement. This study used baseline TRIAD patient survey and chart review data to conduct cross-sectional analyses of associations between processes of care and patient-level cardiovascular and metabolic (cardiometabolic) risk factor levels and other indicators of diabetes care quality.

The TRIAD patient cohort was a stratified random sample of English- or Spanish-speaking adults with diabetes who were living in the community, were not pregnant, and were continuously enrolled and had filed claims with one of the 10 participating health plans during the 18 months before the start of the study. Patients from provider groups with <50 patients with diabetes were excluded.

Participating patients completed a computer-assisted telephone interview or mailed survey between July 2000 and October 2001 and agreed to have their medical records reviewed. Surveys were completed by 11,922 (91%) of eligible respondents. Some potentially eligible patients could not be reached. If patients who were unreachable had the same rate of eligibility as those successfully contacted, the survey response rate would have been 69% (19). Centrally trained reviewers used standardized data collection software to abstract process measures, indicators of comorbidity, and cardiometabolic risk factor levels from medical records. Interrater reliability (κ) for the main quality measures ranged from 0.86 to 0.94. This analysis includes participants who responded to our baseline survey and for whom medical records could be obtained (n = 8,733).

We collected information about process measures of diabetes care, cardiometabolic risk factor levels, patient satisfaction scores, and self-rated quality of diabetes care using patient survey and chart review data sources. Process measures included documentation of A1C, LDL cholesterol, and urine protein tests, as well as foot and dilated eye examinations and receipt of aspirin therapy and influenza immunization during the preceding 12 months. These measures either closely resemble the Alliance (previously DQIP) accountability measures (16,17) or were developed specifically for TRIAD, following DQIP criteria (17,20,21). The first three indicators were determined exclusively from chart review; the remaining four were based on evidence from both the chart review and the patient survey. Chart reviewers abstracted patient chart data for the 12-month period preceding the baseline survey. We then summed the results for individual processes to construct a 0- to 7-point “process score” to reflect the total number of processes performed.

The patient survey collected information about satisfaction with care using items developed for the Consumer Assessment of Health Plans Survey (CAHPS) (22). From these items, we constructed validated scales for perceptions about courteous and helpful office staff (CHS; two questions about respect and helpfulness showed by the office staff) and how well doctors communicate (HDC; four questions about the effectiveness of communication by doctors and time spent by doctors) (22,23). The survey also included items from the CAHPS getting-needed-care scale (GNC; four questions about ease of receiving and choices in primary and specialty care). However, 46% of respondents did not answer the GNC item regarding problems getting a personal doctor or nurse because the health plan did not offer either. Similarly, 48% did not answer the GNC item about problems getting a specialist referral because neither the patient nor physician perceived a need to see a specialist. Therefore, we constructed a modified GNC score that included the remaining items about 1) problems getting care thought necessary and 2) problems with delays in getting needed care because of delays in health plan approval. CAHPS items included in the CHS and HDC scales offer four response options: never, sometimes, usually, or always. We followed recommended CAHPS analysis methods by collapsing the “never” and “sometimes” options into a single category to construct a three-item response (23). We analyzed each scale as the sum of all included items rather than an average item score. Response ranges were 2–6 for CHS, 4–12 for HDC, and 2–6 for the modified GNC scores. We assessed self-rated quality of diabetes care using the following question: “Over the past twelve months, how would you rate the quality of care you received for your diabetes?” Response options included excellent, very good, good, fair, and poor and were coded on a 1- to 5-point scale (poor = 1).

Chart abstractors also collected information about cardiometabolic risk factor levels for the period of 12 months before the patient survey. Measures included levels for last A1C, last LDL cholesterol, last HDL cholesterol, and last systolic blood pressure (SBP) values. We combined individual risk factor levels with information about age, sex, race/ethnicity, and duration of diabetes diagnosis to calculate the UK Prospective Diabetes Study (UKPDS) 10-year predicted probability for a coronary heart disease (CHD) outcome (24–26). This risk engine was developed to predict CHD outcomes in adult patients with type 2 diabetes who do not have preexisting cardiovascular disease. Because 20% of the TRIAD patient cohort had preexisting cardiovascular disease, we used UKPDS scores only to represent a more composite health outcome and not to provide valid predictions of absolute CHD risk in all patients.

The patient survey provided information about participant income, education, a four-level diabetes treatment variable (diet controlled, oral agents only, oral agents and insulin, or insulin alone), and health status as assessed using the Short Form-12 mental and physical component subscores (27). We used chart data about other conditions and current therapy to construct the Charlson index of comorbidity (28,29) and an additional measure of cardiovascular comorbidity that indicates a history of myocardial infarction, stroke, or coronary or carotid revascularization.

We used hierarchical linear models (30) to estimate associations between process-of-care scores and cardiometabolic risk factor levels, satisfaction scales, self-reported quality of care, and UKPDS 10-year CHD risk score. Models were adjusted for demographic and health characteristics and accounted for clustering of observations in provider groups or health plans. Models with A1C as the dependent variable also adjusted for differences in laboratory assays across sites by subtracting the upper limit of normal from the A1C value for each test. From the models, we calculated marginal posterior predictions of intermediate outcomes by process score. Thus, although we performed all analyses using data at the individual patient level, we report associations of process summary scores with average outcomes at the provider-group level. For patients who received primary diabetes care from providers not belonging to a provider group, associations are presented as a group effect at the health plan level. This data structure produced 71 group/plan clusters (68 provider groups plus 3 health plans with direct provider contracting). We reported effects at the group/plan level so that we could examine whether the mean number of care processes for a group or plan were related to average cardiometabolic risk factor levels or to satisfaction or quality of care ratings, while adjusting for various case-mix differences across individual patients.

Models were fit using Bayesian methods (31), as implemented in MLwiN (30). Marginal posterior predictions are expected values for a given outcome obtained from setting the covariates to their mean value (except the process score), setting random effects to 0, and averaging over the posterior distribution.

Eight percent or less of the values for the age, race, sex, treatment, diabetes duration, education, and general state of health variables were missing. Income was missing in 11% of cases. Missing values for all these variables were singly imputed using the transcan function in S-Plus (32). Each variable was predicted as a function of all other variables. Imputation was not performed for the process score or for dependent variables.

The institutional review boards at all translational research centers reviewed and approved the TRIAD study protocol, and all participants provided informed consent.

TRIAD participants encompassed a wide distribution of demographic and health-related characteristics (Table 1), and these characteristics varied considerably across provider groups and health plans (data not shown). Compared with TRIAD participants without available chart records, patients included in this study did not significantly differ with respect to mean duration of diabetes, BMI, or physical or mental component summary subscales of the Short Form-12.

A majority of participants had documentation of five or more of the seven process-of-care measures (Table 2). Documentation of a glycemic assessment (i.e., A1C) and foot examination were most common, whereas aspirin advice or use was less commonly documented. There was only modest variation in mean number of care processes across groups. The mean of group means for the process-of-care summary score was 5.03, with an SD of only ±0.44.

Table 3 depicts the predicted mean change in intermediate outcomes, patient satisfaction scales, and the quality-of-care ratings across all patients of a provider group/health plan with a 1-point increment in the mean sum-of-care processes at the group/plan level (i.e., with an average of one additional care process for each patient in the group/plan).

We found that for each 1-point increment in the mean number of care processes, the predicted mean LDL cholesterol level was 4.51 mg/dl (95% CI 1.46–7.58) lower. We observed no significant associations between higher mean process summary scores and A1C, SBP, or the UKPDS 10-year CHD risk score. However, we did find that a higher mean process score was associated with statistically significantly better scores on patient satisfaction scales (i.e., with CHS, GNC, and HDC) and with self-rated quality of care.

We found that provider groups or health plans that documented a higher average number of seven common diabetes care process measures had patients who, on average, had better levels of some, but not all, measures of diabetes care quality. Although we observed meaningful associations between the sum of simple diabetes process measures and mean LDL cholesterol levels, patient satisfaction, and self-rated quality of diabetes care, we did not observe similar associations with other cardiometabolic risk factor levels. Our findings suggest that these seven simple performance measures also reflect differences in patient-centered indicators of care quality across groups or plans, but they do not capture variations across groups in all dimensions of quality that lead to improved cardiometabolic risk factor control.

One explanation for our findings is that patient satisfaction and self-rated care quality can be influenced more easily than cardiometabolic risk factor levels. Patient perceptions about care may significantly improve in the context of very general strategies to improve the documentation of clinical care processes, particularly if these processes of care lead to a greater level of contact or attention toward the patient. Greater patient contact associated with performing a foot examination, discussing aspirin use, administering influenza vaccinations, or additional visits for a dilated eye examination or laboratory tests to document LDL cholesterol, A1C, and urine protein levels may affect patient perceptions about the comprehensiveness and quality of their care. Moreover, it is also possible that some cardiometabolic risk factor levels are easier to influence than others. For example, it may take longer for general, systems-level quality improvement efforts to have a population-level effect on levels of SBP, A1C, and HDL cholesterol than it does to increase the use of hydroxymethylglutaryl-CoA reductase inhibitors (statins) in the face of elevated LDL cholesterol levels. We had hoped to examine this possibility, but longitudinal analyses of TRIAD provider group-level data are not feasible because a large proportion of groups that were operating during the baseline patient assessment were no longer operating at the time of the patient follow-up assessment 2 years later. As an exploratory analysis, we artificially assigned the 5,432 patients who did have follow-up chart and survey data to the same provider group or plan that they were receiving care from at baseline, and we reran each of the prediction models. These analyses found that the baseline sum of care processes only predicted the HDC scale and self-rated quality of diabetes care 2 years later; we found no associations between the sum of simple diabetes processes of care at baseline and any cardiometabolic risk factor levels at the time of follow-up. It is unclear from these analyses if any possible “real” but delayed effects of group-level quality improvement strategies at baseline would have emerged over time if provider groups had not dissolved.

Our study has some additional limitations. We did not conduct this study to develop an exploratory prediction model that weights individual processes differentially for different outcomes. Instead, we tested an a priori hypothesis wherein documenting more simple processes of diabetes care would be associated with other indicators of higher care quality. Testing this hypothesis involved representing clinical performance by the simple sum of different individual processes of diabetes care. Although considerable evidence supports associations between individual processes and individual risk factor levels, it may be unreasonable to expect that a summary measure comprising multiple process measures should predict any one individual risk factor in isolation. However, we also evaluated associations between the sum of simple processes and more generic quality-related outcomes, including patient satisfaction, self-rated quality of diabetes care, and the 10-year UKPDS CHD risk score, which incorporated several risk factors into one composite measure. We did find significant associations between the sum of simple diabetes processes of care and patient satisfaction and self-rated diabetes care quality. However, the composite CHD risk measure was no more associated with the number of simple processes of care than were the individual levels for A1C, SBP, and HDL cholesterol.

The major driving force for any quality measurement program is to improve medical care to produce better health outcomes. Thus, one natural conclusion from our research is that we need performance indicators that are more strongly linked to health outcomes. Intermediate outcomes have this characteristic but are influenced as much or more by patient factors (e.g., differences in the underlying disease severity of different patient populations) than by the delivery of medical care (33–36). The complexity of efforts required to adjust for differences in disease severity when comparing large groups of patients makes intermediate outcomes far less appealing as a sole method for performance measurement and comparison (33,37). Moreover, if the goal of performance measurement is to identify target areas for efforts to improve medical care, we need to understand something of the relationship between current processes-of-care delivery and the resulting intermediate outcomes. In this context, some form of process measurement may remain necessary to allow us to identify actionable areas for quality improvement (37).

Our study shows that the seven simple process indicators that mirror the accountability measures currently recommended by the Alliance have little association with cardiometabolic risk factor levels. These simple indicators were not initially designed with linkages to risk factor control as a specific objective (15). Furthermore, they do not target those at highest risk or consider valid patient exceptions (e.g., aspirin allergy or lack of desire to consume a daily cholesterol pill) (36,37). We also found little variation in these indicators across individuals, groups, or plans, suggesting that groups and plans have already made good progress toward improvement and that the usefulness of these measures for comparing quality levels between groups or plans is passing (21). Measuring the quality of appropriate medical “actions,” such as appropriate retesting of risk factor levels when they are well controlled and intensifying medical therapies when they are not, will require that we advance to a new generation of performance indicators.

One proposed solution is to link processes-of-care indicators with actionable intermediate outcomes and an assessment of appropriate action by the clinician (36–38). For example, high-quality lipid management for patients with diabetes might be assessed by 1) LDL cholesterol assessment in the past year and 2) initiation or dose adjustment of a lipid medication for individuals with LDL cholesterol values ≥100 mg/dl or documentation of a repeat LDL cholesterol level <100 mg/dl. Such an indicator is more likely to focus attention on a “process” known to improve risk factor levels and exhibits greater physician-level practice variation (38,39). Moreover, “hybrid” performance indicators would offer more information about targets for quality improvement efforts when some patients are identified and treated but still fail to reach recommended treatment goals. Although hybrid performance measures sound attractive when compared with use of either process-of-care indicators or intermediate outcomes alone, more research is needed to determine if they will indeed prove superior to these simpler assessment approaches.

Our work underscores the notion that desirable outcomes of diabetes care are difficult to predict by simply tracking the number of easily obtainable diabetes care processes, which mirror those currently recommended by the National Diabetes Quality Improvement Alliance (16,17). Greater emphasis is needed to move to a next generation of quality measures that capture more distal, actionable processes of care that are likely to reflect both patient-centered measures of care quality and cardiometabolic risk factor control for patients with diabetes.

This study was jointly funded by program announcement no. 04005 from the Centers for Disease Control and Prevention (CDC; Division of Diabetes Translation) and the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK).

The findings and conclusions in this report are solely the responsibility of the authors and do not necessarily represent the official views of the CDC or the NIDDK.

We acknowledge the participation of our health plan partners and that significant contributions to this study were made by members of the TRIAD Study Group.