Although the importance of stakeholder engagement (SE) for patient-centered research is recognized, few studies document SE processes and influence on research outcomes in the diabetes field. We applied a research-informed framework to evaluate the impact of SE on a pediatric diabetes study exploring school nurse perspectives on modern diabetes devices.
We recruited parents of children with type 1 diabetes, school nurses, and diabetes providers. Stakeholders convened virtually every 2 months for 12 months. Goals for SE included input on research materials, interpretation of findings, and future research directions. Processes were assessed using a validated survey. Immediate outcomes included changes to research materials and satisfaction. Secondary outcomes included research efficiency and value (acceptance by community partners).
Each role was represented at every meeting. The majority of stakeholders (>70%) completed the survey at study midpoint and end points. All surveyed indicated that they had received all desired information, shared feedback, and felt valued. Stakeholders were satisfied with the meeting frequency. Participants appreciated learning from each other and expressed enthusiasm for continued research participation. They described their role as one of consultant rather than research team members. SE resulted in five additional interview questions. Nearly 70 comments added to the interpretation of qualitative themes. Findings were published within 12 months and recognized by the state school nursing organization.
SE was well received and led to meaningful changes in content and dissemination of a diabetes study. A systematic approach to evaluating SE can increase scientific rigor and reproducibility and contribute to best practices for SE in diabetes research.
The Patient-Centered Outcomes Research Institute (PCORI) defines stakeholders as patients or other individuals with a vested interest in health-related research that may affect them (1). Stakeholder engagement (SE), which is central to patient-centered research, considers stakeholders as equal partners, integrating them into all phases of the research process (2). SE may have varied intensity depending on the nature of the study, ranging from garnering advice to including stakeholders as full-fledged coinvestigators. Despite increasing support for stakeholder-engaged research, there is little evidence supporting optimal approaches to SE (3–5). Specific challenges include a lack of consistent terminology and reporting guidelines for describing outcomes resulting from SE in research.
As with other fields, effective SE may add value to patient-centered research in endocrinology and diabetes to improve patient care and outcomes. Stakeholders are myriad in pediatric diabetes research and include adolescents and young adults with diabetes, their parents, health care providers, other professionals (e.g., payers and pharmacists), leaders of organizations (e.g., the American Diabetes Association), and other caregivers in the community (e.g., school nurses). In recent years, pediatric diabetes research involving stakeholders has been used to set research priorities (6), determine research outcomes (7,8), and design education/decision aids (9,10). However, few studies describe the processes used for SE (11,12) or formally evaluate the impact of SE in diabetes research. To increase the scientific rigor of SE, Ray and Miller (13) proposed a novel, research-informed framework to guide investigators in the planning, evaluation, and reporting of research on SE. We sought to apply the framework described by Ray and Miller to comprehensively evaluate SE in a study examining school nurse perspectives of modern diabetes devices in the school environment.
Research Design and Methods
Context
We conducted a stakeholder-engaged, qualitative research study using semistructured interviews to explore the experiences, attitudes, and current practices of school nurses related to modern diabetes devices. The results of this study have been published elsewhere (14). In parallel, we conducted research evaluating SE using the Ray and Miller framework (Figure 1) (13). The purposes for stakeholder involvement in this qualitative study included garnering input on methodology (e.g., revision of interview guide), recruitment strategy, interpretation of findings, and planning for future research. In line with PCORI-defined best practices (1,15), we sought perspectives from parents of children with type 1 diabetes who use devices, school nurses, and diabetes care providers. All activities were led by the principal investigator (PI) with guidance from experienced senior researchers. The University of Pittsburgh Institutional Review Board deemed this study exempt (PRO1901004).
Processes
We identified potential stakeholders representing our broader region (Western Pennsylvania) through preexisting relationships with local school nurse leadership and clinical colleagues. The PI contacted potential stakeholders to review the proposed role and gauge interest. We aimed to have at least two representatives for each role. Meetings occurred every 2 months using a Web-based video conferencing platform (Zoom Video Communications) with individual phone calls for those unable to attend. Stakeholders received an agenda and research materials in advance of all meetings.
At the initial meeting, the PI reviewed expectations, ground rules for respectful discussion, and the anticipated time commitment. Subsequent meetings mirrored the research phase with discussions about 1) research materials, 2) brief didactics (e.g., on federal and state laws for the management of diabetes in schools), 3) interpretation of findings, and 4) future research directions. The PI facilitated discussions of agenda items and the decision-making process among stakeholders. Stakeholders were encouraged to ask questions and share their perspectives, facilitating colearning. Because the meetings were virtual, the PI took notes in real time using a shared screen to promote transparency. We distributed a follow-up e-mail message, including notes and recommendations to the group, and stakeholders were encouraged to send comments between meetings. Stakeholders received a one-time financial compensation for their contributions.
Evaluation and Outcomes
SE processes were evaluated along nine domains as outlined by Ray and Miller (13). Recruitment and composition were assessed by recruitment rate and the balance of participants. Representativeness and value of contributions were assessed by meeting attendance, the variety of contributions, and incorporation of feedback. Stakeholder perspectives on their expectations, joint-decision making, frequency of communications, colearning, and transparency were assessed anonymously with a validated, open-ended survey at the midpoint and end points of the study (Table 1) (13,16). Because the questions were open-ended, we tabulated yes/no responses and included representative quotations from stakeholders. We defined immediate outcomes as changes to research materials and stakeholder satisfaction. Intermediate outcomes were defined as research efficiency (time to study completion), research value (acceptance by community partners), and stakeholder empowerment. Long-term outcomes were not assessed in this study.
Question . | Domain . |
---|---|
1. Have you received all of the information that you would like to receive about the project? | Co-learning |
2. How has your experience so far compared with your expectations? | Expectation-setting |
3. How would you describe your role on this project to others? | Expectation-setting |
4. For each of the following types of communication, please indicate if you would like more, the same, or less:
| Frequency |
5. Have you been able to provide all of the feedback that you would like to share about the current research project? | Decision-making |
6. Has the project team adequately responded to the feedback you have shared? | Transparency |
7. How can we improve our ability to incorporate your knowledge and recommendations into current and future projects? | Transparency |
8. Please share any other comments you have about this experience and how we can improve it. | NA |
Question . | Domain . |
---|---|
1. Have you received all of the information that you would like to receive about the project? | Co-learning |
2. How has your experience so far compared with your expectations? | Expectation-setting |
3. How would you describe your role on this project to others? | Expectation-setting |
4. For each of the following types of communication, please indicate if you would like more, the same, or less:
| Frequency |
5. Have you been able to provide all of the feedback that you would like to share about the current research project? | Decision-making |
6. Has the project team adequately responded to the feedback you have shared? | Transparency |
7. How can we improve our ability to incorporate your knowledge and recommendations into current and future projects? | Transparency |
8. Please share any other comments you have about this experience and how we can improve it. | NA |
Results
Evaluation of Processes
Seven stakeholders (100% of those approached) were recruited, representative of the previously identified roles. The committee included two parents, two school nurses, and three diabetes care providers (two advanced practice providers and one diabetes care and education specialist). Two of the providers also had type 1 diabetes. Six meetings, each 1 hour long, were held over the course of 1 year; each role was represented at every meeting. Five stakeholders (71%) completed the anonymous survey at the midpoint and six (86%) at the end point; responses at both time points were similar. Based on feedback from the midpoint survey, we added routine e-mail updates between meetings and sent verbatim meeting notes to “member check” discussion points (17). A summary of the evaluation of SE processes, including end point survey responses, is provided in Table 2.
Domain(s) . | Measure . | Summary of Findings . |
---|---|---|
Recruitment and composition | Recruitment rate; balance of different roles | • 100% of those approached agreed to participate • Representation of parents of a child with type 1 diabetes (n = 2), school nurses (n = 2), and diabetes providers (n = 3) |
Representativeness | Meeting attendance; contributions from different roles | • Each meeting had at least one representative from each role • Everyone contributed at least one comment during meetings |
Expectation-setting | Stakeholder survey | • Majority describe the role as “consultant” or as providing their opinion. One self-described as a “stakeholder” • 100% felt the experience met or exceeded their expectations |
Decision-making | Stakeholder survey | • 100% felt they had contributed all desired feedback • Participants reported that they “appreciated having my voice heard” |
Frequency | Stakeholder survey | • At midpoint, two participants wanted increased e-mail communication • At end point, participants recommended no change to frequency of meetings, emails, or telephone calls |
Valuing of contributions | Recommendations incorporated into research | • All suggestions were accounted for in revising research materials • Comments on themes enriched the discussion in the published manuscript |
Co-learning | Stakeholder survey | • 100% felt they had received all important information • Participants felt meetings were “openly collaborative” |
Transparency | Stakeholder survey | • At midpoint, one participant recommended sending verbatim notes • 100% felt that their input was addressed • Participants reported feeling that their “input was taken seriously” and that they received “clear and detailed reports” |
Domain(s) . | Measure . | Summary of Findings . |
---|---|---|
Recruitment and composition | Recruitment rate; balance of different roles | • 100% of those approached agreed to participate • Representation of parents of a child with type 1 diabetes (n = 2), school nurses (n = 2), and diabetes providers (n = 3) |
Representativeness | Meeting attendance; contributions from different roles | • Each meeting had at least one representative from each role • Everyone contributed at least one comment during meetings |
Expectation-setting | Stakeholder survey | • Majority describe the role as “consultant” or as providing their opinion. One self-described as a “stakeholder” • 100% felt the experience met or exceeded their expectations |
Decision-making | Stakeholder survey | • 100% felt they had contributed all desired feedback • Participants reported that they “appreciated having my voice heard” |
Frequency | Stakeholder survey | • At midpoint, two participants wanted increased e-mail communication • At end point, participants recommended no change to frequency of meetings, emails, or telephone calls |
Valuing of contributions | Recommendations incorporated into research | • All suggestions were accounted for in revising research materials • Comments on themes enriched the discussion in the published manuscript |
Co-learning | Stakeholder survey | • 100% felt they had received all important information • Participants felt meetings were “openly collaborative” |
Transparency | Stakeholder survey | • At midpoint, one participant recommended sending verbatim notes • 100% felt that their input was addressed • Participants reported feeling that their “input was taken seriously” and that they received “clear and detailed reports” |
Midpoint and end point survey responses were similar; only end point responses are presented unless otherwise specified.
Evaluation of Immediate Outcomes
As a result of SE, five questions were added to the interview guide regarding 1) concerns about liability, 2) challenges specific to substitute nurses, 3) desired clinical information from diabetes care providers to aid school-based care, 4) mechanisms for sharing information with diabetes care providers and parents/legal guardians, and 5) desired collaboration with diabetes care providers. The stakeholders agreed with the planned recruitment strategy targeting a representative statewide sample through the Pennsylvania Association for School Nurses and Advanced Practitioners. In reviewing transcripts, stakeholders felt the a priori codebook resonated with their experiences.
In reviewing the interview findings, a total of 68 comments were provided in discussion of emerging and final themes: 22 (32%) from diabetes care providers, 16 (24%) from school nurses, and 30 (44%) from parents. In these comments, the stakeholders identified shared experiences with the quotations from interview transcripts and discussed important contextual issues that affected the interpretation of the themes. Specifically, they discussed how children may benefit from devices in school (e.g., vibration alarms on continuous glucose monitoring [CGM] devices allow for more discretion), the variable degree of parental involvement with the nurse (e.g., planning and communication with the school nurse both before and during the school year are critical), concerns related to liability (e.g., some parents make more frequent dose adjustments, which may conflict with school orders), pressures on school nurses (e.g., competing demands during the school day may affect their ability to monitor CGM data), and limitations of available resources (e.g., a need for more technology-specific training but lack of clarity on who would provide or pay for this training). Diabetes care providers remarked that it was helpful to hear that school nurses wanted to be more involved in diabetes care, which the school nurse stakeholders echoed. They identified potential strategies to maintain an open dialogue among parents, school nurses, and diabetes care providers such as a secure, teams-based platform. They jointly identified future research priorities, including a focus on enhancing communication among caregivers and training for school nurses to optimize school-based diabetes care for youths.
Overall, stakeholders reported satisfaction with the experience, including the organization of meetings, facilitated discussions, and responses to their feedback. Stakeholders felt valued and that “all team members’ input was considered equally.” Stakeholders also highlighted the benefit of learning from one another; one individual stated that the experience “has given me the opportunity to appreciate the perspectives of others.” Another indicated, “I wish I had … even more time and effort to contribute to this project, which I know [has] such an important impact for kids with diabetes.” At the study’s completion, 100% of the stakeholders agreed to continue in this role.
Evaluation of Intermediate Outcomes
After the initial stakeholder meeting, recruitment and interviews began within 1 month, and the results were published within 12 months (14). Findings were accepted for oral presentation at the statewide school nursing association annual conference (Pennsylvania Association for School Nurses and Advanced Practitioners 2020 annual meeting). Stakeholder empowerment emerged in discussion points and survey comments. Stakeholders emphasized that this research area was “crucial,” as families have a wide variety of experiences at school. The stakeholders expressed the desire to participate in future research in this area. One wrote, “I’m excited to continue to hear about the findings and work together to provide valuable advice to our patients, nurses, and health care providers.” In discussion regarding legal concerns with school-based care, some stakeholders discussed advocacy at the state level to encourage legislative changes such as clarifying the school nurse role with regard to diabetes devices.
Discussion
We describe here the processes, evaluation, and impact of SE conducted in parallel to a qualitative study examining school nurse perspectives of caring for children who use modern diabetes devices. SE directly contributed to the methodology through additions and clarifications to the research materials and interpretation of findings as part of the qualitative analysis. Specifically, five additional interview questions recommended by stakeholders yielded key findings that were emphasized in the final themes of the qualitative study about the school nurse role with regard to CGM, managing parent expectations, and existing barriers to communication with diabetes centers. Stakeholder guidance on recruitment strategies assisted with efficient participant enrollment. The discussion of emerging and final themes was enriched by stakeholders sharing their personal experiences, which provided additional context that supported and strengthened our findings. Furthermore, in review of our findings and discussion of remaining challenges, stakeholders collectively found agreement on two future research priorities, which were central to guiding our team’s subsequent work. Our study adds to the limited body of literature describing the utility of SE to augment the quality, transparency, and relevance of pediatric diabetes research.
Within endocrinology and related fields, SE contributes to patient-centered research by encouraging a holistic approach that accounts for the values of those ultimately affected by this research. Broadly, research on chronic disease, including diabetes or other endocrinologic diseases, is well suited to SE (18). Similar to other conditions, the management of pediatric diabetes necessarily relies on collaboration among parents, multidisciplinary care providers, and other community supports in the care of children. These clinical relationships can translate to effective alliances focusing on patient- and caregiver-centered research priorities for diabetes and other chronic diseases of childhood. The framework presented herein can serve as a guide for other investigators in conducting and reporting on SE in diabetes.
Research on SE reports on the processes used to partner with stakeholders and subsequent implications for research, rather than the outcomes of the parent study itself. Although SE is increasingly recognized as a funding priority (19), there is a paucity of literature evaluating the impact and effectiveness of SE. An early assessment of SE in PCORI-funded studies by self-report of the investigators and partners found that SE resulted in meaningful refinement to the research (20). Across these studies, SE occurred in various research phases (e.g., intervention design, outcome measurement, recruitment, analysis, and dissemination) and to differing extents, although the overall perception was that stakeholders exerted at least a moderate influence.
Although helpful, descriptively reporting on SE without clear measures makes it difficult to compare outcomes of SE across studies and to clarify best practices. Challenges persist in understanding how to report on SE across different types of studies and populations with consistent terminology for outcomes (13). Clear, validated methods are needed to examine the rigor of various SE approaches to enhance research, augment transparency, promote reproducibility, and ensure that research findings are relevant to end users (2,5). The framework applied in this study could serve as a standard framework for researchers to systematically report on their processes and outcomes with SE (13).
Overall, our stakeholders valued their experience engaging in research and learning from one another. Stakeholders felt that their input was valued and described their experience as “openly collaborative.” We have retained partnership with all members of this committee and are working on additional research projects. One member of our SE committee is included here as a coauthor (C.M.), showcasing the cooperative nature of this work.
A key principle for SE is ensuring clear understanding of roles and expectations (21). Although surveyed participants agreed that the project met or exceeded their expectations, the majority still described their role as consulting or simply sharing opinions, rather than as members of the research team. Enhancing language around partnership may empower stakeholders to see themselves as part of the scientific team, rather than as advisors, better reflecting growing interest nationally in team science (22). In future work, we can foster SE by generating research questions together, assigning specific roles within the study, or considering training to optimize the research literacy of community partners (23).
Our study has some limitations. Our stakeholder group included parents but did not include adolescent youth; this was intentional given the target population of the qualitative study (school nurses caring for elementary and middle school students). Not all stakeholders completed the survey; however, feedback during the meetings echoed survey findings. Although the qualitative data provided some insight into the stakeholders’ perceived impact on the research, this concept could be examined in more depth using an SE assessment matrix to understand whether their current level of engagement mirrors their desired participation (24). Next steps will include increasing the roles represented on the stakeholder committee and refining our processes based on their feedback. We will continue SE for future research, examining parent and provider perspectives about school-based care, establishing training goals for school nurses, and developing interventions to optimize school-based diabetes care.
Future work within this field involves standardizing methods for research on SE such as further validating stakeholder questionnaires, establishing consistent definitions for intermediate- and long-term outcomes, and identifying minimum requirements for reportable outcomes. In addition to methodology, studies may examine how SE practices vary across diverse populations and different types of community partners. SE in pediatric research is a unique area requiring attention, as partnering with both youth and their caregivers adds another layer of complexity. Finally, implementation of this framework by research teams may provide a comparable standard for assessing the degree of SE in research and establish best practices for collaborative team science that acknowledges the crucial role of stakeholders.
Research on SE is an emerging field. Literature describing outcomes from SE is limited but expanding, particularly within the disciplines of diabetes care and endocrinology. We evaluated the processes and impact of SE on a qualitative research study using an established framework. A systematic approach to reporting research on SE in the medical literature can contribute to building evidence-based practices and increase the rigor and reproducibility of this field in patient-centered outcomes research in the area of pediatric diabetes.
Article Information
Acknowledgments
The authors thank the members of the stakeholder advisory group for their contributions to this research.
Funding
This research was funded by the UPMC Children’s Hospital of Pittsburgh Cochrane Weber Endowment Award.
Duality of Interest
No potential conflicts of interest relevant to this article were reported.
Author Contributions
C.A.M., T.M.K., I.M.L., and E.M. contributed to the conceptualization and design of the study. Under the mentorship of E.M. and T.M.K., C.A.M. led all stakeholder meetings and conducted the surveys. C.M. participated as a stakeholder. All authors contributed to the interpretation of findings and writing process and read and approved the final manuscript. C.A.M. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Prior Presentation
Findings from this study were presented at the International Society for Pediatric and Adolescent Diabetes virtual annual conference, 16–18 October 2020.