Validity and Reliability of an Instrument for Assessing Health-Related Quality of Life and Treatment Preferences: The Insulin Delivery System Rating Questionnaire

The IDSRQ was developed through a three-step procedure. First, the authors reviewed existing instruments and articles comparing different insulin delivery systems. Next, two focus groups were conducted with diabetes patients. Participants were asked about the advantages and disadvantages of the various insulin delivery systems they had used. Based on these sources and our own experiences with diabetes patients, we constructed a set of items and obtained feedback from several diabetes experts. The revised version of the instrument was used for the current pilot study.

All participants were adults with insulin-treated diabetes. Participants were recruited from the practices of 12 physicians across the country. Each physician was asked to recruit up to 25 patients who were using an insulin pump (any make or model; continuous subcutaneous insulin infusion [CSII] group) and another 25 patients who were taking multiple (three or more) daily insulin injections using syringe or pen (multiple daily injections [MDI] group). These two insulin delivery systems were chosen because both represented relatively intensive forms of treatment, and previous research had indicated differences between the groups on many patient-reported outcomes. The study protocol called for physicians to recruit consecutive patients who were at least 18 years of age, had type 1 or type 2 diabetes for at least 1 year, were using an insulin pump (CSII) or taking MDI, and were able to read English well enough to fill out the IDSRQ questionnaire on their own.

Patients were contacted by the physician or a staff member and asked to participate in the study. Those who expressed interest were given a packet containing an explanation of the study and a copy of the questionnaire, along with a stamped self-addressed envelope for returning the questionnaire directly to the researchers.

Participants were asked to provide a name and address so another (identical) questionnaire could be sent to them 2–3 weeks after receiving the initial questionnaire to assess test-retest reliability of the IDSRQ. The follow-up questionnaire had an identification number so that it could be linked to the first questionnaire filled out by the subject. A return envelope was provided.

This protocol was approved by the Human Subject Research Committee of Loyola College (Baltimore, MD).

In addition to demographic questions, the IDSRQ asked about frequency of blood glucose monitoring, what insulin delivery system is being used now, and what system was used immediately before the current system. (A copy of the instrument is available upon request.) Seven multi-item subscales were generated, one for each section of the questionnaire. Three subscales (treatment satisfaction, treatment interference with daily activities, and clinical efficacy) asked questions specific to the respondent’s insulin delivery system. Three subscales (diabetes worries, social burden, and psychological well-being) asked more general questions. One subscale assessed overall preference for the insulin delivery system. The score for each item was a metric ranging from 0 for the lowest response option to 100 for the highest response option, with equal distance between response categories. Scale scores were computed as the mean of the completed items.

Stem question: “How satisfied are you with your current insulin delivery system?” Response categories (from high to low): completely satisfied, very satisfied, somewhat satisfied, or not at all satisfied. Sample items: how convenient it is, how easy it is to take insulin, how painful it is, embarrassment when using it, how complicated it is to use, and uncertainty about getting the amount of insulin intended.

Stem question: “How much does your current insulin delivery system interfere with the following?” Response categories (from high to low): a lot, some, a little, or not at all. Sample items: wearing the clothes you want to, getting a good night’s sleep, eating when you want, exercising as much as you want, engaging in sexual activities, and taking care of yourself when traveling.

Stem question: “How good is your current delivery system in helping you with the following?” Response categories (from high to low): excellent, very good, good, fair, or poor. Sample items: getting good blood glucose control, keeping your blood glucose stable, avoiding doctor visits for poor control, avoiding hospital visits for poor control, and avoiding weight gain.

Stem question: “How often do you worry about the following?” Response categories (from high to low): all the time, frequently, sometimes, rarely, or never. Sample items: getting complications, unpredictable blood glucose, being home alone, and travel away from home.

Stem question: “How often are the following true for you?” Response categories (from high to low): all the time, frequently, sometimes, rarely, or never. Sample items: friends/family worry about your blood glucose levels, friends/family have to help you take care of your diabetes, your doctor does not think you take good care of your diabetes, and you have to see your doctor often to get help with your diabetes care.

Stem question: “How often do you have the following feelings about yourself and your life?” Response categories (from high to low): all the time, frequently, sometimes, rarely, or never. Sample items (eight positive items and seven negative items; negative items are reverse scored): energetic, in a good mood, in control of your body, anxious, overwhelmed, and self-conscious.

This scale used four questions, each with its own response categories. Two items specifically represent not just preference but degree of preference (desire to switch to another system and comparison to previous system). Question: “Would you like to switch to another insulin delivery system?” Response categories (from low to high): definitely yes, probably yes, probably not, or definitely not. Question: “How would you compare your current insulin delivery system to your previous system?” Response categories (from high to low): current much better, current a bit better, both about the same, previous a bit better, or previous much better. The latter question was asked only of participants who had used an insulin system in the past that was different from that being used currently and included an instruction to compare the current system to the system used immediately before the current system. Two items represent additional dimensions of overall evaluation (satisfaction and recommendation). Question: “Overall, how satisfied are you with your current insulin delivery system?” Response categories (from high to low): completely, very, somewhat, or not at all. “Would you recommend your current insulin delivery system to others?” Response categories (from high to low): definitely yes, probably yes, probably not, or definitely not.

The overall treatment preference measure was analyzed in two ways: with and without the item comparing current treatment strategy with prior treatment strategy. This item is most appropriate when the comparator is the same for all respondents or there are two groups in which the current and comparator systems are reversed (e.g., CSII versus MDI in one group and MDI versus CSII in the other group).

Reliability of subscales was measured by Cronbach’s α and item analysis. Test-retest analysis used the intraclass correlation coefficient and Student’s t tests of mean differences. Floor and ceiling effects were measured by the number of respondents receiving the minimum and maximum scores, respectively. Data quality was assessed by the completeness of responses. Validity was assessed by comparing the difference in means between CSII and MDI groups using ANOVA with and without controlling for age, sex, and type of diabetes. Effect sizes were calculated as the difference in SD units.

Key determinants of overall preference were identified by hierarchical multiple regression analysis controlling for age, sex, and type of diabetes. Multiple regression analysis was used to determine which variables made an independent contribution to the prediction of overall preference and to eliminate the “halo effect,” in which there is a tendency to rate something high or low in all respects. Incremental variance in overall preference explained by the IDRSQ subscales was assessed by comparing r² before and after entry of these measures. Degree of mediation for overall preference was determined by comparing coefficients for the difference between CSII and MDI before and after entry of the IDRSQ measures.

The criterion for statistical significance was set at P < 0.05, two-tailed for all analyses. All analyses were conducted using SPSS statistical software (version 11.5; SPSS, Chicago, IL).

Sample characteristics are shown in Table 1. Most participants (71.1%) used CSII at present. The sample was primarily patients with type 1 diabetes (72.1%), women (52.8%), and individuals of middle age (mean age 46.4 years); the mean duration of diabetes was 22.6 years. Most subjects (88.8%) monitored their blood glucose three or more times per day. Of those taking injections, the average number of daily injections was 4.2 (61.9% took four injections). Most injection users used a syringe exclusively (54.4%) and only 5.3% used a pen exclusively; the rest of the subjects used a combination. Of those using CSII, most subjects (77.9%) had used a syringe alone as their insulin delivery system just before switching to CSII.

Table 2 shows the descriptive statistics for the IDSRQ measures. Measures generally had acceptable psychometric properties. The α reliability coefficients ranged from 0.67 to 0.92 (median 0.82). The reliability for the overall preference scale was increased to an acceptable level (>0.70) when the treatment comparison item was dropped; this is likely due to the fact that all subjects were not using the same treatment strategy for comparison. Item analysis indicated that α could be increased slightly (<0.025) by dropping one item from four of the measures.

Floor effects (percent with minimum score) ranged from 0 to 10% (median 0%), and ceiling effects (percent with maximum score) ranged from 0 to 18% (median 4%). Only 1.65% of responses were missing; only the treatment comparison item had more than 10 missing responses (18 of 22 missing responses on this item were because the respondent did not have a prior treatment strategy against which to compare the current treatment strategy). Numbers of respondents with no missing data on a particular measure ranged from 173 (88%) to 195 (99%), with a median of 182 (92%).

Of the 197 initial respondents, 114 (57.8%) completed the follow-up questionnaire and were used in the test-retest analysis. Test-retest reliability ranged from 0.67 to 0.94 (median 0.88). Four measures showed a statistically significant shift in the mean score over time, but all shifts were <0.25 SD units.

Table 3 shows significant differences between treatment strategies on all measures except psychological well-being. Group differences on these measures ranged from 0.3 to 1.4 SD units (median 0.9). The differences were substantial for all items from the treatment preference measure. More injection users (58.2%) than CSII users (17.2%) expressed desire to switch to another insulin delivery system. More CSII users (93.6%) than injection users (56.4%) were very or completely satisfied overall with their insulin delivery system. More CSII users (83.6%) than injection users (24.6%) stated that they would definitely recommend their insulin delivery system to others. More CSII users (97.1%) than injection users (65.7%) reported that their current system was better than their prior treatment strategy.

Table 4 reports the results of multiple regression analysis. The first row of the table shows that the difference in overall preference between CSII and MDI was significant for the total sample after controlling for age, sex, and type of diabetes (and for both type 1 and type 2 participants after controlling for age and sex). In the full sample, factors assessed by IDSRQ subscales accounted for 27% of the variance in overall preference (32% for type 1 diabetes and 21% for type 2 diabetes), and mediated 53% of the difference in overall preference between CSII and MDI (60% for type 1 diabetes and 52% for type 2 diabetes). When each IDSRQ measure was entered into a model separately, each had a significant association with overall preference (results not shown). When all were entered into one model together, treatment satisfaction, clinical efficacy, and psychological well-being had significant independent associations with overall preference for all respondents and for those with type 1 diabetes. Only clinical efficacy was significantly associated with overall preference for those with type 2 diabetes, although the coefficients for treatment satisfaction, interference, and social burden would have been significant if the type 2 diabetes sample were as large as the type 1 diabetes sample. When a separate analysis was performed only for those who were CSII users, treatment satisfaction, clinical efficacy, and psychological well-being had independent associations with overall preference (results not shown).

The questionnaire used in this study differs in several important respects from the Diabetes Quality of Life (24) questionnaire, the most frequently used measure of HRQOL in people with diabetes. First, it assesses treatment satisfaction and impact of treatment specific to the insulin delivery system and is, therefore, likely to be more sensitive to specific but important differences in these factors among delivery systems (22,23,25). Second, the IDSRQ incorporates a broad range of measures, including perceived clinical efficacy and psychological well-being, neither of which the Diabetes Quality of Life assesses in detail. Third, the IDSRQ provides a multidimensional measure of overall preference, incorporating not only a retrospective comparison with the previously used delivery system but also a prospective measure of interest in changing systems and a willingness to recommend the current system to others (which may play an important role within the diabetes patient community). Although the IDSRQ was used in this study to assess MDI and CSII, it is designed to be used with any insulin delivery system (e.g., inhaled insulin) and to permit comparisons of various systems (e.g., pen versus syringe, one pump versus another pump, etc.).

The IDSRQ demonstrated good validity by detecting significant differences between treatment groups on all but one domain. Although the observed differences between groups were statistically significant, that does not necessarily indicate that they are “important” or “meaningful.” Unlike objective clinical measures such as HbA_1c, there are no established clinical criteria for assessing the size of the differences on these measures. However, a recent review of self-report measures (26) identified 0.5 SD units as the criterion for a “minimally detectable difference,” the smallest difference that a person would be able to detect. This difference, which is a “moderate” effect size (27), was observed for six of the seven IDSRQ measures. Therefore, the statistically significant differences between treatment groups identified by the IDSRQ reflect subjectively meaningful differences. That is, patients can recognize the differences involved, and those differences can influence decisions they make about their treatment.

The IDSRQ had acceptable psychometric properties (internal consistency, floor/ceiling effects, and test-retest correlation). Although the test-retest analysis showed statistically significant shifts over time in responses for some measures, the shifts were relatively small (under 0.25 SD units, less than half the size of the minimally detectable difference). However, further research with larger samples is required to assess construct validity. Specifically, confirmatory factor analysis should be used to determine whether all items load significantly and primarily on hypothesized dimensions.

The IDSRQ seems to be useful in identifying the factors that contribute to users’ preferences for different insulin delivery systems. Several earlier studies have attempted to identify the factors that influence patients’ preference for CSII over MDI, the two insulin delivery systems assessed in this study (5,13–18). The authors of those studies offered conjecture about these factors, suggesting that any observed difference between CSII and MDI could be (one of) the key difference(s). The IDSRQ assessed these factors (and others) and found that there were differences between CSII and MDI on most. But until the current study, there had been no explicit attempt to determine which of the observed differences actually contribute to patient preference. Not all factors on which CSII and MDI differ significantly make independent contributions to the preferences for the different insulin delivery systems, as we have shown. Our analysis found that two IDSRQ domains (treatment satisfaction, clinical efficacy) made independent contributions to patients’ overall preference for their insulin delivery system as evidenced by a significant difference between groups, combined with a significant association with preference. Therefore, the results of previous studies have been confirmed using a more rigorous methodology. Future research examining determinants of treatment preference should use this methodology rather than offering conjecture about the possible contribution of these factors.

The major limitation of this study is the fact that the samples were not designed to provide a generalizable comparison between CSII and MDI. The samples are not representative of the populations using CSII and MDI or of all persons treating their diabetes with insulin. The physicians who recruited patients were chosen because they were favorable toward CSII and could, therefore, provide a substantial number of CSII patients to facilitate the comparison between treatment groups. Also, CSII was not necessarily offered to all their patients, so those who started on CSII may represent a sample that initially was more positively predisposed toward this insulin delivery system. However, the current study was designed for the purpose of instrument validation; we sought to create a group Δ so we could test the instrument’s ability to detect differences between treatment groups in patient-reported outcomes. The results indicate that we were successful. But studies designed to compare patient outcomes for different insulin delivery systems will need to use equivalent samples, i.e., those created through random assignment.

Although the IDSRQ was sensitive to differences between groups, this study was not designed to demonstrate sensitivity to within-subject change over time. Also, we do not know whether it will be sensitive enough to be able to detect differences between other insulin delivery systems. However, the preliminary results reported here suggest that it will be worthwhile to pursue these applications.

We did not determine whether the differences between insulin delivery systems in perceived clinical efficacy were associated with differences in objective outcomes such as HbA_1c levels or episodes of severe hypoglycemia. However, others have shown that satisfaction with care is associated with better glycemic control (28). Perceptions of one’s insulin delivery system could well be related to one’s willingness to initiate and maintain intensified treatment and, consequently, related to improved clinical outcomes. Therefore, perceived efficacy may be an important factor in its own right, independent of its relationship to objective treatment efficacy.

In summary, this pilot study suggests that the IDSRQ is a valid and reliable instrument for assessing patient perceptions of their insulin delivery systems. Anyone interested in using this instrument may obtain a copy directly from the first author (there is no charge for its use). Given the importance of insulin in the management of diabetes and the increasing availability of alternative insulin delivery systems, we believe that further research should be conducted to ascertain the potential of this instrument.

This research was funded by an unrestricted educational grant from Medtronic MiniMed.