The ePedsQL™ in Type 1 and Type 2 Diabetes: Feasibility, reliability, and validity of the Pediatric Quality of Life Inventory™ Internet administration

Participants were children aged 5–18 years (n = 92) and parents of children aged 2–18 years (n = 93) diagnosed with type 1 or type 2 diabetes from a hospital-based pediatric endocrinology clinic. There was 1 child (1.1%) aged 2–4 years, 7 children (7.5%) aged 5–7 years, 32 children (34.4%) aged 8–12 years, and 53 children (57.0%) aged 13–18 years. Participant characteristics are shown in Table 1.

The sample of healthy children was taken from the previously conducted PedsQL initial field test (6) and a State's Children's Health Insurance Program evaluation (13). The sample was randomly matched by age, sex, and race/ethnicity to the diabetic sample utilizing SPSS statistical software random-sample case selection command (SPSS, Chicago, IL). The average age of the 429 boys (50.9%) and 414 girls (49.1%) was (means ± SD) 12.01 ± 3.02 years. There were 504 (59.8%) white non-Hispanic subjects, 164 (19.5%) Hispanic subjects, 123 (14.6%) black non-Hispanic subjects, 11 (1.3%) Asian/Pacific Islanders, 20 (2.4%) American Indian or Alaskan Natives, and 21 (2.5%) classified as “other.”

The 23-item PedsQL Generic Core Scales encompass 1) physical functioning (eight items), 2) emotional functioning (five items), 3) social functioning (five items), and 4) school functioning (five items) and were developed through focus groups, cognitive interviews, pretesting, and field-testing protocols (5,6).

The PedsQL scales are comprised of parallel child self-report and parent proxy–report formats. Child self-report includes children aged 5–7, 8–12, and 13–18 years. Parent proxy–report includes children aged 2–4 (toddler), 5–7 (young child), 8–12 (child), and 13–18 (adolescent) years and assesses parents' perceptions of their child's HRQOL. The items for each of the forms are essentially identical, differing in developmentally appropriate language, and are first- or third-person tense. The instructions ask how much of a problem each item has been during the past 1 month. A five-point Likert response scale is utilized across child self-report for children aged 8–18 years and parent proxy–report (0 = never a problem, 1 = almost never a problem, 2 = sometimes a problem, 3 = often a problem, and 4 = almost always a problem). To further increase the ease of use for the young child self-report (ages 5–7 years), the response scale is reworded and simplified to a three-point scale (0 = not at all a problem, 2 = sometimes a problem, and 4 = a lot of a problem), with each response choice anchored to a happy-to-sad-faces scale.

Items are reverse scored and linearly transformed to a 0–100 scale (0 = 100, 1 = 75, 2 = 50, 3 = 25, and 4 = 0), so that higher scores indicate better HRQOL. Scale scores are computed as the sum of the items divided by the number of items answered (this accounts for missing data). If >50% of the items in the scale are missing, the scale score is not computed. This accounts for the differences in sample sizes for scales reported in the tables. The Physical Health Summary Score (eight items) is the same as the Physical Functioning Scale. To create the Psychosocial Health Summary Score (15 items), the mean is computed as the sum of the items divided by the number of items answered in the Emotional, Social, and School Functioning Scales.

In developing the ePedsQL, we sought to create a user-friendly interface that was comparable in format with the paper-and-pencil child self-report and parent proxy–report versions of the PedsQL 4.0 Generic Core Scales with several color enhancements. The interface was designed such that each PedsQL scale was presented on a separate screen and respondents were not able to select more than one answer for each item. If one or more items on a PedsQL scale were not answered when the respondent attempted to proceed to the next screen, a pop-up message appeared that read, “You have not answered all the questions on this page. Are you sure you want to continue?” Respondents could either select “yes,” which would allow them to proceed to the next screen or “no,” which would take them back to the screen that contained one or more unanswered items. The back button on each screen allowed respondents to return to the previous screen to view or change their answers before submitting their final responses.

Before the field test, the ePedsQL was pretested with 15 patients aged 5–18 years and 19 parents of patients aged 2–18 years from the pediatric endocrinology clinic. Qualitative interviews were conducted with participants after they completed the ePedsQL in order to assess acceptability and preference for the format of the software. Children and parents were asked a number of open-ended questions including, “Did you like the look of the computer questionnaire?,” “Were the directions for the computer questionnaire clear?,” and “Were the words on the computer questionnaire easy to see?” Children and parents were also asked to provide suggestions for improving the look and usability of ePedsQL. Interviews lasted ∼1 h and were audiotaped. Detailed complete notes were taken by the research nurse during the interviews. Three of the authors thoroughly reviewed these notes and transcripts in order to develop a content analysis of the major themes.

Overall, the interviews revealed that children and parents found the ePedsQL user-friendly and visually appealing. Nonetheless, based on feedback from the interviews, some minor formatting changes were made. For example, the font on the ePedsQL was changed from Times New Roman (size 10) to Arial (size 14) to make the words easier to read. Given that some parents reported not reading the directions, we bolded and underlined parts of the directions so that they would stand out. Children and parents both suggested that the use of more color would enhance the look of the ePedsQL. As a result, we added green next and back buttons and yellow highlighting that appeared when respondents scrolled over or selected a response choice. The letters of the term “PedsQL” were colored in blue, green, red, and yellow to further enhance the color presentation as suggested by the children. The rest of the instrument contained shades of blue and white backgrounds with black lettering consistent with the paper version.

For the field test, children and parents each completed the paper-and-pencil version of the ePedsQL Computer Use and Satisfaction Questionnaire designed for this study. The Computer Use and Satisfaction Questionnaire contained questions regarding preference for mode of administration and ease of use of the software platform. A five-point Likert response scale was utilized across child self-report for children aged 8–18 years and parent proxy–report (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree). A three-point Likert response scale (0 = strongly disagree, 2 = neither agree nor disagree, and 4 = strongly agree) was utilized for young child self-report (ages 5–7 years). The Computer Use and Satisfaction Questionnaire also contained an item asking children and parents to report their frequency of computer use. Response options across child self-report and parent proxy–report included “never,” “once a month,” “once a week,” “once every few days,” and “every day.”

A paper-and-pencil and Internet version of the PedsQL Family Information Form (6) was completed by each parent during the field test. The Family Information Form contains demographic information including the child's date of birth, sex, race/ethnicity, and parental education information.

A randomized crossover design was utilized. Participants completed both the ePedsQL and the PedsQL paper-and-pencil version during one session. Using random numbers generated by SAS, the child was randomly assigned to complete either the ePedsQL or the PedsQL paper-and-pencil version. The parent simultaneously completed the alternative mode of administration version (i.e., when the child first completed the paper-and-pencil version, the parent first completed the Internet version and vice versa). After the child and parent completed the first version of the PedsQL, they engaged in an interversion distraction activity. Specifically, parents and children completed the 18-item PedsQL Multidimensional Fatigue Scale (7,14). Parents also completed the paper-and-pencil version of the Family Information Form during the interversion interval. The interversion interval lasted ∼5–7 min and was terminated when the parent had completed the paper-and-pencil version of the Family Information Form. Upon completion of the interversion activity, the child and parent completed the PedsQL in the alternative mode of administration. This randomized crossover design with a brief interversion interval was modeled after two equivalence testing studies in pediatric and adult electronic health assessment (11,12). Finally, the child and parent completed the Computer Use and Satisfaction Questionnaire after completing the other measures.

For both modes of administration, children aged 5–18 years were given a specific PedsQL age-appropriate form depending on their age (5–7, 8–12, and 13–18 years). For children aged 2–4 years, only parent proxy–report measures were obtained. The research assistant helped the younger children (aged 5–7 years) complete the measures and was present to assist the older group as needed. For both the ePedsQL and the PedsQL paper-and-pencil administration, starting and finishing times were recorded by the research assistant.

Participants completed the instruments in a patient examination room in the pediatric endocrinology clinic, either while waiting for the physician or after the child's appointment with the physician. Examination rooms were equipped with web-enabled desktop computers. Each child/parent pair received a $10 gift certificate for their participation in the study. This research protocol was approved by the institutional review board at the Scott and White Memorial Hospital and Clinics.

Feasibility was determined by calculating the percentage of missing values for child self-report and parent proxy–report across modes of administration (15). The mean duration (in minutes) for completing the ePedsQL and the PedsQL paper-and-pencil version was compared using paired-samples t tests (12). Feasibility was further assessed by calculating the percentage of children and parents who agreed or strongly agreed with the following statements on the ePedsQL Computer Use and Satisfaction Questionnaire: “The computer questionnaire was easy for me to use,” “It was easy to learn to use the computer questionnaire,” “I was able to work fast using the computer questionnaire,” and “If I could choose, I would rather fill out the computer questionnaire rather than the paper-and-pencil questionnaire.”

Internal consistency reliability for both modes of administration was determined by calculating Cronbach's coefficient α (16). Scales with reliabilities of ≥0.70 are recommended for comparing patient groups, while a reliability criterion of 0.90 is recommended for analyzing individual-patient scale scores (17,18). Feldt's method was utilized to test the equivalence of α coefficients across mode of administration (19).

One of the main advantages of a randomized cross-over design is that it allows for the evaluation of whether the same individuals provide equivalent answers across mode of administration (12). In this context, score equivalence refers to the similarity in responses across mode of administration (12). At the group level, we compared PedsQL mean scores across mode of administration using paired samples t tests for all participants who completed the Internet and paper-and-pencil modes of administration. Additionally, effect sizes were calculated to determine the magnitude of any differences across modes of administration (20). Effect size, as used in these analyses, was calculated by taking the difference between the paper-and-pencil mean and the Internet mean divided by the paper-and-pencil SD. Effect sizes for differences in means are designated as small (0.20), medium (0.50), and large (0.80) in magnitude (20).

We assessed the individual agreement of PedsQL scores across mode of administration using intraclass correlation coefficients (ICCs) (21). ICCs are designated as ≤0.40 poor to fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 good agreement, and 0.81–1.00 excellent agreement.

Construct validity was assessed utilizing the known-groups method (22), which compares scale scores across groups known to differ in the health construct being investigated. Using independent sample t tests, we compared the ePedsQL scores of the present diabetic sample with the PedsQL scores (paper and pencil) of a matched sample of healthy children. Based on previous research with the PedsQL version 4.0, we hypothesized that children with diabetes would demonstrate lower HRQOL than healthy children (10). To determine the magnitude of the differences between the healthy and diabetic samples, effect sizes were calculated (20). Effect size, as utilized in these analyses, was calculated by taking the difference between the healthy sample mean and the diabetic (Internet) sample mean divided by the healthy sample SD. Statistical analyses were performed using SPSS.

The majority of children (91.2%) and parents (96.7%) strongly agreed or agreed that the ePedsQL was easy to use. Additionally, 90.2% of children and 95.7% of parents strongly agreed or agreed that it was easy to learn to use the ePedsQL, 92.4% of children and 97.8% of parents strongly agreed or agreed they were able to work fast using the ePedsQL, and 82.4% of children and 83.7% of parents strongly agreed or agreed that if they had the choice they would prefer to complete the ePedsQL rather than the paper-and-pencil administration of the PedsQL 4.0 Generic Core Scales.

On average, the ePedsQL took children 2 min and 59 s (SD 1 min and 45 s) to complete, while the mean completion time for the PedsQL paper-and-pencil version was 2 min and 15 s (SD 58 s) for children (P < 0.01). On average, the ePedsQL took parents 3 min and 38 s (SD 2 min and 14 s) to complete, while the mean completion time for the PedsQL paper-and-pencil version was 2 min and 21 s (SD 45 s) for parents (P < 0.001). It should be noted that child and parent reports of their frequency of computer use on the ePedsQL Computer Use and Satisfaction Questionnaire were significantly correlated with the ePedsQL mean completion times (child, r = −0.33, P < 0.01; parent, r = −0.30, P < 0.05), indicating that more frequent computer use was associated with shorter completion times. This finding may account in part for the greater variability found in completion times for the electronic mode of administration compared with the paper-and-pencil version. Another possible explanation for the longer time required for the Internet mode of administration is that each of the four scales was presented on a separate screen, requiring time to navigate to a different screen for each scale, while the paper-and-pencil version had all four scales on one page.

For child self-report, the ePedsQL resulted in a similar percentage of missing-item responses (0.76% for all scales) compared with the PedsQL paper-and-pencil administration (0.71% for all scales) (P > 0.05). For parent proxy–report, the ePedsQL resulted in a slightly smaller statistically nonsignificant percentage of missing-item responses (0.37% for all scales) compared with the PedsQL paper-and-pencil administration (0.56% for all scales) (P > 0.05).

Internal consistency reliability coefficients for the ePedsQL and the PedsQL paper-and-pencil administration are presented in Table 2. All child self-report and parent proxy–report scales on the ePedsQL exceeded the minimum reliability standard of 0.70 required for group comparisons, while the total scale scores for both child self-report and parent proxy–report approached or exceeded the reliability criterion of 0.90 recommended for analyzing individual patient scores. There were no statistically significant differences between α coefficients across modes of administration.

Table 2 demonstrates the differences between PedsQL scores across modes of administration. For each PedsQL scale and summary score, there were no statistically significant differences across mode of administration for child self-report and parent proxy–report. All effect sizes were essentially nil. Table 2 presents the ICCs between the modes of administration. ICCs were in the excellent agreement range for each scale and summary score.

Table 2 demonstrates the differences between our sample of children with diabetes (Internet administration) and the matched healthy sample (paper-and-pencil administration). Children with diabetes and their parents reported statistically significant lower HRQOL than healthy children, with the exception of the Social Functioning Scale. Most effect sizes were in the medium range.

This study presents the measurement properties of the ePedsQL Internet mode of administration for the PedsQL 4.0 Generic Core Scales in type 1 and type 2 diabetes. The analyses support the feasibility, reliability, and validity of the ePedsQL as a child self-report and parent proxy–report HRQOL measurement instrument for diabetes. The ePedsQL is the only empirically validated electronically administered pediatric HRQOL instrument to span this broad age range for child self-report and parent proxy–report while maintaining item and scale construct consistency.

Items on the ePedsQL had minimal missing responses, suggesting that children and parents are able to provide good quality data regarding the child's HRQOL, utilizing an Internet mode of administration. The ePedsQL self-report and proxy-report internal consistency reliabilities exceeded the minimum standard of 0.70 for group comparison, while the ePedsQL total scale score met or exceeded an α of 0.90, which is recommended for individual patient analysis (17), making the total scale score suitable as a summary score for the primary analysis of HRQOL outcomes in clinical trials and other group comparisons.

The ePedsQL preference ratings are consistent with other studies in the literature. For example, in a study using the short-form 36, 83% of patients preferred the electronic versus the paper-and-pencil mode of administration (11), although another study found that the time to completion was faster with the short-form 36 paper-and-pencil versus electronic administration (23). As more families have access to and experience with the Internet, it might be anticipated that the preference and speed of completion for Internet administration will increase.

The present findings have several potential limitations. Information on nonparticipants and participants' socioeconomic status were not available, which may limit the generalizability of the findings. However, parental education level was obtained, indicating that most parents were high school graduates or had attended college but were not college graduates. The interval between modes of administration was short; however, this design and interval length was modeled after previously published electronic versus paper-and-pencil mode-of-administration studies with adults and children with ∼5-min interversion intervals (11,12). As pointed out by the investigators in these studies, including a 5-min interversion diversionary activity mitigates in part remembrance of the first assessment, as well as the possible contamination effects that may occur during longer interversion intervals (e.g., changes in symptoms over time). The sample size of children with type 2 diabetes was small. The comparisons between the healthy children and diabetic sample of the present sample used different modes of administration. However, given the findings from the present study, it would not be expected that the ePedsQL data from a healthy sample would differ significantly from the paper-and-pencil administration findings. The satisfaction measure was designed for the purposes of the present investigation and may overestimate the participants' satisfaction ratings. However, the satisfaction ratings are similar to other published studies utilizing electronic administration. Further research is required to determine the feasibility of ePedsQL administration without the presence of trained personnel, such as at home or at a kiosk. Finally, these findings with the ePedsQL were only for the Generic Core Scales. Measurement research with the PedsQL Diabetes Module is indicated utilizing the ePedsQL mode of administration.

Preparation of this manuscript was supported by an intramural grant from the Texas A&M University Research Foundation. Data collection was supported by an intramural grant from the Scott and White Memorial Hospital Research Foundation.

The authors express appreciation to Michael Sparks for creating the ePedsQL interface.