Making Diabetes Electronic Medical Record Data Actionable: Promoting Benchmarking and Population Health Improvement Using the T1D Exchange Quality Improvement Portal

Most T1DX-QI centers (88%) use the Epic EMR system (26), while the remainder (12%) use the Cerner system (27). Even among T1DX-QI centers using the same EMR system, there is considerable variability in documentation practices, especially in collecting diabetes- related data variables. Many data points of specific interest in type 1 diabetes care do not exist in the standard EMR software package. Therefore, data collection methods must be custom-created by each center over many years, resulting in wide variations of parameter definitions, completeness, and types of reports.

For this reason, data standardization was a vital and challenging early step to allow T1D Exchange to calculate comparable metrics across all centers. The T1D Exchange requires centers to map data to network- defined data standard specifications, which, in some cases, requires centers to realign how they define and collect data or add new variables. T1DX-QI data specifications consist of more than 120 unique variables across seven data files: Patients, Providers, Encounters, Observations, Conditions, Medications, and Diabetes (Table 1, Supplementary Material). Files one through six comprise the “core” files, consisting of many standard EMR variables. The Diabetes file contains information on adverse outcomes, glucose monitoring, insulin plan, and patient glucose readings. Limited protected health information (dates and five-digit zip codes) is shared with the T1D Exchange; centers share unique patient identifiers to support longitudinal tracking.

The data specifications were created in 2018 by the T1DX-QI Data Science Committee, composed of principal investigators and data analysts. This group guides the optimization of the QI Portal system, which is updated biennially to align with evolving standards of care and best practices. The committee also oversees biennial changes to data specifications, reviewed and approved in quarterly committee meetings.

Patient-level data from the local EMR databases are de-identified. These data are transmitted into the QI Portal and contribute to T1DX-QI population health research (16,17). Centers send data submissions via secure file transfer (28), where they are processed and uploaded to a business intelligence system (29) feeding the QI Portal (Figure 1A). Data are stored securely in the United States; only users who have a need for the data have access to the QI Portal, which has two-factor user authentication to protect privacy and security. All centers comply with their local institutional review board regulations to use protected health information. All centers also complete business associate agreements with T1D Exchange, highlighting data use expectations.

The data-mapping process (Figure 1B) is usually completed in 3–12 months based on the local center’s infrastructure and skilled staffing. The first step involves setting up a secure file transfer protocol (28) to transfer files to the T1DX-QI coordinating center. Once this connection is established, centers submit one comma-separated values file at a time on a subset of data (e.g., 2 months of encounters). T1DX-QI technical staff complete quality checks to evaluate whether the data file is formatted per the data specifications. If a file does not meet the expectations, feedback is provided and the process is iterated until there are no issues with the file. This process is repeated for each of the seven files. Once all files have passed review, an additional check is conducted on a historical dataset (typically covering 5 years of data).

The next phase, data validation, focuses on data accuracy. Centers validate against T1DX-QI calculations to ensure data quality and verify that the data are accurate (external validation). Then back-end data processing is verified to produce expected values for metrics in the QI Portal (internal validation). After validation, centers automate monthly data transfers, contributing data to the QI Portal and EMR database. As of March 2022, 22 endocrinology centers have been mapped and regularly contribute data to the QI Portal.

Initially launched in 2018, the QI Portal has undergone multiple feature upgrades (Figure 2). Early iterations allowed users to display QI run charts (30) of center-specific key outcome metrics against the collaborative average (e.g., median A1C). Recent versions have expanded functionality and includes features such as a dashboard (Figure 3), the benchmarking Compare function (Figure 4), a Reports section for more detailed center-level data analysis and statistical process control charts (Figure 5) (31), and a Resource Library for QI insights, best practices, and improvement success stories (Figure 3) (31). There are 12 available measures in the QI Portal (Table 2), including median A1C, percentage of continuous glucose monitoring (CGM) use, percentage of insulin pump use, and frequency of diabetes-related ketoacidosis (DKA) events. QI Portal users can also custom-design charts with options for selecting time ranges (month and year) and filtering charts by patient demographics, including age, sex, race/ethnicity, insurance, and primary language (Figure 5). Notably, the QI Portal incorporates a health equity feature that allows users to disaggregate outcomes by race/ethnicity, displaying multiple outcomes by racial/ethnic identity and supporting the center’s efforts to identify opportunities to increase equity (32). Version 3.1 expanded the user population by allowing non–data-mapped centers access to the Compare and Resource Library features, increasing engagement with the QI Portal and allowing non–data-mapped users limited access to relevant features. Most recently, version 3.4 released additional metrics (time in range and social determinants of health) and expanded health equity features to allow users to examine disaggregated data by sex, language, and insurance.

T1D Exchange tracks engagement with the QI Portal as the percentage of data-mapped centers with at least one user logging in monthly. An average number of users per month is calculated per calendar year to track shifts in user engagement over time using Google Analytics. The QI Portal displays clinical outcomes with Laney-p′ charts. T1D Exchange also invites users to complete user feature interviews to collect user feedback. User feature interviews are also shared in the monthly T1DX-QI newsletter.

T1DX-QI centers use the QI Portal as a tool for functions that are vital to support an LHS (33,34). The QI Portal addresses the lack of standardized data elements and definitions, allowing center-to-center comparison of clinical outcomes across highly variable and noninteroperable EMR systems. The QI Portal unlocks information-dense EMR systems to enable centers to use their data, particularly type 1 diabetes–specific information, outside of a typical EMR data package for the purposes of benchmarking and population health management.

The QI Portal is designed for four user types: 1) endocrinologists/clinical champions, 2) QI coordinators/analysts, 3) QI/population health researchers, and 4) clinical and administrative team leaders. T1DX-QI centers can investigate center-specific, patient-level, and aggregate data through customizable charts compared with other T1DX-QI centers’ aggregated outcomes. T1DX-QI centers benefit from network-wide and center-specific demonstrations of data to contribute to the growth and learning across network centers. Specifically, QI Portal users benefit from benchmarking, population health management tools, health equity insights, QI coaching, and network learning and sharing of best practices (35). The sections below describe how QI Portal (version 3.1) supports users in each element.

Understanding the quality of clinical care processes and services is essential to health care QI (36). QI Portal data allow centers to understand their current system state and use real-time measures to support iterative improvement cycles. This function is crucial to users such as QI coordinators/analysts and clinical champions. The QI Portal allows centers to use clinical outcomes analytics to understand whether tests of change have the desired effect. Users can chart the 12 QI Portal metrics against specified time frames and filter for specific patient populations. Control charts demonstrate whether significant process shifts have occurred, in which case users can annotate charts to describe the clinical changes made (Figure 5).

The QI Portal data platform currently receives data from 22 U.S. type 1 diabetes centers’ EMR systems. Thorough data-mapping to T1DX-QI specifications provides an opportunity to benchmark key clinical outcomes across highly variable EMR systems. QI Portal users have multiple options for benchmarking. First, users receive a snapshot of T1DX-QI ranking across critical metrics (Figure 3). Users can customize dashboard views to select the top five metrics of interest, comparing their rank against other T1DX-QI data-mapped centers and the previous month’s outcomes. This benchmarking feature is valuable to users in the clinical/administrative team leader category who are interested in, for example, understanding their center’s overall performance. Users can also compare aggregate center data on the Compare tab (Figure 4).

T1DX-QI has been using data to contribute to health equity improvement (16,37–39). Real-world data are crucial to LHS health equity work (40,41). Just as in QI work, timely, center-specific data provide information on how well a process is currently operating, identify improvement opportunities, and validate whether tests of change were successful. Data can also be a tool for buy-in and urgency building. Users can filter data by selected patient demographics such as race/ethnicity, sex, primary insurance, language, and other categories (Figure 5). Users can also display data stratified by race/ethnicity and visually display and save reports to share with team members (Figure 5) (39). This feature is valuable across all four QI Portal user types to understand center performance for QI and work related to health equity. An upgrade is underway to allow users to stratify outcomes by the remaining demographics/equity indicators.

The ability of the QI Portal to integrate EMR data into population-level summaries and outcome trends is a valuable tool for QI and population health research users. The QI Portal allows users to quickly visualize several metrics and summaries on their type 1 diabetes population, with customizable reporting templates. Summary statistics provide demographic breakdowns, and users can save reports on key outcome metrics for future reference. Additionally, by using unique patient identifiers assigned for this endeavor (not patients’ medical record numbers), users can download patient lists to identify individual patients included (or not) in charts and analyses.

The T1DX-QI strives to create a culture of collective learning and growth. QI Portal users can see top-performing centers and connect with them directly to share tips for improving outcomes. QI Portal users can access a library of resources, including change packages and T1DX-QI case studies (Figure 3). Change packages describe the methodology implemented to improve clinical or operational processes (e.g., increasing depression screening or building QI capacity) (42,43). Case studies summarize successful QI projects completed by T1DX-QI centers. These resources benefit all users, particularly endocrinologists/clinical champions and QI analysts/coordinators.

T1DX-QI tracks center engagement across various metrics, including QI Portal use. T1D Exchange staff track the percentage of T1DX-QI centers and the number of monthly users who log into the QI Portal. Figure 6 shows trends in user engagement between January 2020 and December 2021 and a 31% increase in the percentage of centers logging into the QI Portal monthly from 2020 to 2021. The number of unique QI Portal users more than quadrupled (4.8 times) in that time period. Over the same period, the number of individual patients with data in the QI Portal increased from ∼25,000 to >45,000 as more centers completed data-mapping.

Efforts to increase user engagement include diversifying QI Portal functionality, expanding the user population, and featuring user interviews in the monthly T1DX-QI newsletter. Users have reported sharing control charts, comparison reports, and population-level summaries across QI teams and institutional leadership.

To quote one clinical champion, “I have most frequently used the [QI] Portal to pull A1C graphs to communicate global trends to our executive stakeholders, as well as personnel in the center. The [QI] Portal makes this task much faster than running ad hoc internal reports and generating the graphics on my own.” Users have also reported enjoying immediate access to data summaries and rankings across T1DX-QI centers. One center staff member said, “I use the dashboard during our meetings with the principal investigators (PIs) to show how we are doing. Our PIs have even used this feature to show our clinical staff, so they know how we are working toward improvement.” Additional quotes from user interviews are described in Supplementary Table S1.

Mapping clinical data from 22 U.S. type 1 diabetes centers and harmonizing these data within the QI Portal supports an LHS to improve practices. Centers complete a comprehensive data-mapping process to map local EMR systems to T1DX-QI data specifications, allowing comparison of key outcomes across participating centers. This novel platform enables T1DX-QI centers to access clinical data for QI, benchmarking, population health analysis, and sharing of best practices. Although published literature cites the role of LHS technology and data considerations (2), fewer than 40% of LHSs reviewed by Enticott et al. (6) were categorized as being real-world data–enabled.

The QI Portal addresses many LHS challenges previously described in the literature, including challenges in linking data, noninteroperable EMR systems, lack of standardization of data elements and definitions, and lack of robust data aggregation or analysis tools (7–10). The QI Portal illustrates how a single tool can support multiple centers’ needs within an LHS beyond just benchmarking. Additionally, unlike other platforms (2), the QI Portal is a tool to address social justice through features that facilitate examining issues related to health equity. This platform plays a pivotal role in supporting the T1DX-QI LHS to improve outcomes for people with type 1 diabetes.

Platforms like the QI Portal possess potential value beyond LHSs; accountable care organizations and other provider-based risk models require real-world data, benchmarking, and population health insights to improve care quality and value (44,45). This account of the QI Portal development can be a reference for future population health data platforms to improve patient care across a spectrum of systems and models.

One strength of this work is that the QI Portal is a one-of-a-kind platform contributing to practice advancement in 22 endocrinology centers across the United States, collectively caring for >45,000 individuals with type 1 diabetes. The QI Portal is a unique tool to unlock EMR data to transform clinical practice. Across variable EMR systems, the QI Portal allows centers to directly compare outcomes among centers and subsets of patients. The QI Portal provides each center with real-world data access through monthly submissions and updates.

Limitations of this work include variability in data fields that centers have had to define for themselves (e.g., pump and CGM naming conventions). The T1DX-QI Data Science Committee collects examples and templates of flowsheets and best-practice documentation workflows to share across centers and support standardization. Another limitation is that T1D Exchange does not receive all data variables from all centers. Centers continue to enhance data collection to increase the number of variables submitted. Additionally, there is no use case (hypothetical scenario for use) for patients or parents of patients unless they are active members of a center’s QI team. T1D Exchange plans to incorporate patient feedback and experience into future versions of the QI Portal. Finally, the data-mapping process requires a substantial investment of time and resources. Among the most challenging obstacles is obtaining sufficient priority from institutions’ information technology teams to undertake the tedious mapping, validating, and transferring of data.

New features for QI Portal version 3.5 are currently in development to increase user ease of use and experience and prepare to add 2023–2025 T1DX-QI measures. The next major QI Portal release (version 4.0) is being planned and will be informed by interviews with users and other stakeholders, a review of T1D Exchange strategic priorities, and demands in the broader field of type 1 diabetes care. All options will be evaluated for impact and value for the QI Portal to continue improving type 1 diabetes care and outcomes.

The authors acknowledge Sanjeev Mehta for his contributions to T1DX-QI data specification development. They also acknowledge all members of the T1DX-QI Collaborative, patients, and information technology and data teams across the T1DX-QI network and data partners. The authors thank the Leona M. and Harry B. Helmsley Charitable Trust for funding the T1DX-QI Collaborative and the QI Portal. None of the funders were involved in the development or review of the manuscript or the decision to publish.

M.C. is the chief medical officer of Glooko, and he has received research support from Abbott Diabetes Care and Dexcom. O.E. is an advisor to the Medtronic Diabetes Health Equity Board, and his organization (T1D Exchange) has received compensation for his time; through his organization, he has also received support for research projects from Dexcom, Eli Lilly, and Medtronic Diabetes. No potential conflicts of interest relevant to this article were reported.

A.M. researched the data and wrote the first draft. A.M., O.E., M.C., and G.T.A. developed the outline and critically reviewed the manuscript. O.E. conceived the concept for the manuscript, researched data, and critically revised the manuscript. M.C. and G.T.A. collaborated with others to build the QI Portal’s data specifications. All other authors critically revised the manuscript and approved the final version. A.M. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.