Clinical Implementation of Autonomous Artificial Intelligence Systems for Diabetic Eye Exams: Considerations for Success

Key to successful implementation is that stakeholders within the system must be engaged at the inception of the program, while it is still in the planning phase. These stakeholders include physician champions in the deploying department (i.e., primary care or endocrinology clinic), from the departments of ophthalmology/optometry, and from the information technology (IT) teams and staff. Stakeholders need to be comfortable with the safety, efficacy, and equity of the system, including such issues as mitigation of racial bias, data usage and cybersecurity, reimbursement, and value-based care.

The primary users of the system (i.e., primary care or endocrinology teams) should identify a champion within their practice or division to facilitate and oversee implementation of the AI technology. This champion should be responsible for informing and educating providers about the AI system, its functionality, and the workflow around its use. It is important to be considerate of the current workflow and to work with providers and clinic staff to optimize the new workflow that will incorporate implementation of diagnostic AI (29).

Getting buy-in from the institution’s ophthalmology/optometry departments, if available, is helpful in evaluating which AI system to choose, providing technical camera support and camera operator expertise, and setting up scheduling to ensure there is an efficient referral pathway for patients identified as having potentially blinding disease.

It is also important to engage the IT team, who will play a crucial role in helping to set up the camera in the clinic, connecting it to the network, assisting with any local software issues, and connecting the AI camera to the EMR system. Connecting the local IT team with the vendor IT team will also facilitate a smooth startup. The experience of implementing other systems worldwide has highlighted the importance of multidisciplinary team collaboration in the integration of AI into routine workflows (29).

Autonomous AI systems (i.e., nonmydriatic fundus cameras with assistive AI for image quality feedback and autonomous diagnostic AI—what the FDA refers to as “medical devices”) can be used as a standalone, independent system. In this case, a formatted report of the diabetic eye exam results can be uploaded into the EMR system or printed and scanned into the medical record. PDF documents of the results can be printed for patients or e-mailed to patients.

At our institution, the autonomous AI system is fully integrated through HL7 (Health Level Seven) with our EMR system in a secure manner. A provider places an order for an autonomous AI diabetic eye exam in the EMR system. This order is then released to the camera software, and the patient’s name is automatically added to the worklist on the camera software. The camera operator then clicks on the patient’s name from the worklist and proceeds with obtaining the retinal images for the diabetic eye exam. Once the exam is complete, the results are generated as a PDF report within 30–60 seconds, automatically uploaded/pushed to the EMR system, and stored under the “Media” tab of the EMR. The actual images also remain on the camera and can also be pushed to an ophthalmic data management platform if necessary. Many ophthalmic data management platforms exist. Currently, our institution uses Forum Viewer, v. 4.2.4.15 (Zeiss, Dublin, CA).

Once the order for the autonomous diabetic eye exam is placed and completed, the best practice alert (BPA) notification for DR screening on the diabetes dashboard is fulfilled. This is important, particularly in adult clinical practices where HEDIS-MIPS measures for care gap closure should be met (17).

Of note, the autonomous AI system reformats the retina images from the fundus camera in DICOM (Digital Imaging and Communications in Medicine) format, which can then be saved in any imaging database. In our system, the images are stored on the camera and manually pushed into our ophthalmic data management platform from the camera.

A tabletop nonmydriatic fundus camera requires a designated space in the clinic setting with enough room for the camera and its associated computer to sit on an adjustable tabletop, as well as room for the patient chair for imaging. Space requirements would be minimal for handheld cameras with autonomous AI technology, and although they are used widely around the world with varying image quality, these devices currently are not approved by the FDA (30). For best imaging results, it is ideal to place the camera in a dimly lit room and away from any windows to avoid glare from outside light. If there is a window in the room, we recommend darkening shades to keep much of the light from entering the space with the camera.

The computer and camera software can be operated by a technician or operator (who should be a high school graduate, per FDA labeling for the device). The AI system of LumineticsCore (formerly IDx-DR) has an image quality algorithm that determines whether the images are of sufficient quality for the diagnostic algorithm to process and provide a result (10). The camera software also guides the operator on image acquisition to further ensure sufficient images for interpretation. In the pediatric study, a learner operator curve was created and demonstrated that it took ∼40 patient eye exams for an operator to reach a consistent average of 6 minutes to complete image acquisition (20). In subsequent use of the camera with new operators, we have recommended at least 20 practice eye exams before imaging patients for clinical care. Additionally, in the real-world clinical practice setting where there may be many medical assistants (MAs) or nurses triaging patients, we recommend training a few designated camera operators who can quickly build their imaging experience by using the camera and software regularly, as opposed to investing time and resources for training all of the MAs and nurses in a clinic.

In the first pivotal trial conducted using LumineticsCore in adults with diabetes, the overall imageability was 96.1%, and 23.6% of adults required pharmacologic dilation to obtain an AI system result (10). The pivotal trial of the EyeArt Automated DR Detection System, v. 2.1.0, also demonstrated high imageability of 87.4% without dilation and up to 97.4% with a dilation protocol (11).

Our experience in pediatrics is that no pharmacologic dilation is required, and simply dimming the lights causes spontaneous pupil dilation that is sufficient to obtain images for AI system interpretation (20). In contrast, our experience with the adult population led to two main findings. First, the diagnosability rate in real-world deployment could be lower than that of clinical trials. In an analysis of our data, we identified patient factors, including older age, current smoking, and having type 1 diabetes, that were associated with nondiagnostic results. Using factors associated with nondiagnostic results, we have developed a model that can predict nondiagnostic results for future design of a novel predictive dilation workflow, in which patients most likely to benefit from pharmacologic dilation are dilated a priori to improve workflow efficiency and optimize completion of the diabetic eye exam (31).

Several factors leading to insufficient (i.e., nondiagnosable) images have been identified in studies using retinal images in both teleretinal networks and autonomous AI. In a study of adults by Liu et al. (8), small pupils, cataracts, or other media opacities led to suboptimal retinal image quality. Inconsistent image quality in combination with comorbidities has made real-world implementation challenging in adults (32,33). In our pediatric study, we identified image quality issues of blurring, edge artifacts, dark images, and diffuse reddish hue as causes of insufficient images. The real-world use of autonomous AI systems has demonstrated that images of inadequate quality can lead to false-positive results by the AI system (13,34,35). Thus, if the operator obtains images with these artifacts, the images should be retaken before processing the images through the autonomous AI algorithm (Figure 1).

Developing a streamlined workflow for integrating point-of-care (POC) diabetic eye exams in any busy clinical practice is important for success. We suggest considering three factors: 1) identifying patients who need a diabetic eye exam, 2) integration of AI in the clinic workflow, and 3) establishing referral routes for abnormal AI exams (Figure 2).

There are several potential methods for identifying patients who are due for a diabetic eye exam, and this process may depend on the practice’s EMR system and/or clinical workflow. Institutions using Epic may have a diabetes dashboard that provides “care gaps” for patients with a diabetes diagnosis or best practice alerts (BPAs) that inform providers when a diabetic eye exam is due. If these systems are not available, a manual chart review of upcoming patients can identify who is due for diabetic eye exams.

The clinical workflow of a multidisciplinary diabetes clinic is complex, so one option is to notify patients in advance of their visit that a POC diabetic eye exam with immediate results is available at their appointment. Once a patient arrives in the clinic, there are several possibilities for the workflow: 1) the eye exam can be performed by the MA as part of the check-in process, 2) the eye exam can be performed after the patient checks in but while he or she is waiting to be seen by the provider, or 3) the eye exam can be performed at the end of the visit. The limitation to performing the eye exam at the end of the visit is that this process would require the provider to wait and return to the patient once the eye exam is complete and results are available. Educating patients about the importance of the diabetic eye exam and the importance of having a follow-up eye care visit when referable eye disease is identified at the point of care has been shown to improve adherence with eye care follow-up (8,36).

Providing abnormal results at the point of care instead of in a deferred manner is associated with higher adherence to follow-up eye care. In a randomized controlled trial using AI-assisted DR screening in a low-resource setting, the researchers found that providing an abnormal result at the point of care instead of 3–5 days later was associated with higher rates of follow-up with an eye care provider (37). Additionally, a prospective cohort study performed using AI screening also demonstrated improved adherence to follow-up eye care when abnormal results were provided at the point of care with education on the importance of eye care follow-up (8).

To facilitate follow-up for patients with AI-positive eye exams, develop a referral route by providing a list of local ophthalmologists/optometrists to help patients find an eye doctor with whom to schedule an appointment. In health care systems with optometry or ophthalmology practices, the diabetes clinic or primary care office should consider setting up a direct referral system such that referral orders can be placed and patients can schedule their follow-up eye care visit on their own, or the eye clinic staff can contact patients to schedule the follow-up visit, or follow-up eye care visits can be scheduled before patients leave their diabetes appointment.

Widespread adoption of autonomous AI for DED screening requires alignment of incentives of different stakeholders within a particular health care ecosystem. The existence of an established screening program such as the Singapore Integrated DR Program in Singapore or the National Health Service Diabetic Eye Screening Program in the United Kingdom could facilitate the adoption of AI by providing the necessary IT infrastructure, referral workflow, and regulatory framework. However, in the absence of widespread, well-coordinated screening programs, the alignment of financial incentives is of paramount importance. For example, as previously mentioned, a new CPT code 92229 was established in the United States in 2021 specifically for the billing and reimbursement of autonomous AI exams for DED. This code is for “point of care autonomous analysis that uses innovative autonomous AI technology to perform the interpretation of the eye exam, without requiring that an ophthalmologist interpret the results.” The Centers for Medicare & Medicaid Services considers CPT code 92229 to be a diagnostic service reimbursable for Medicare and provides a definitive way for primary care providers to perform and be reimbursed for autonomous AI testing (19,38).

The ultimate goal of deploying autonomous AI screening for DED is to improve compliance with diabetic eye exams because regular screenings can lead to early diagnosis, timely intervention, and prevention of irreversible vision loss. In our first deployment in pediatrics, we measured the baseline screening rate at 49%, and this rate improved to 95% after implementation of autonomous AI diabetic eye exams (20). At Johns Hopkins Medicine, we initially deployed autonomous AI for diabetic eye exams at four adult primary care clinics in 2020. We compared the change in the DED screening adherence rate from 2019 (pre-AI deployment) to 2021 (post-AI deployment) between the sites with autonomous AI screening (AI sites) and sites with standard- of-care screening (non-AI sites). From 2019 to 2021, the odds of compliance increased by 87% in the AI sites (odds ratio [OR] 1.87, 95% CI 1.74–2.01) compared with 10% in the non-AI sites (OR 1.10, 95% CI 1.05–1.15) (P <0.001) (39). Future analysis of patient-level data will use multivariable logistic regression models to identify at-risk groups and streamline screening and referrals to optimize compliance and early identification of DED.

Additionally, acceptability by both providers and patients as key stakeholders in this process is important for the success of implementation and dissemination. A limited body of literature on this topic suggests that ∼55% of patients believe that AI will improve their health care, while voicing concerns related to patient choice, potentially increased health care costs, bias in the data source, and data security (40,41). Although there is general enthusiasm in the medical community for AI in health care, there is focused attention on the explainability of AI, as well as ensuring ethics and equity in the AI algorithm and its implementation (13,14,42).

Many studies have demonstrated that underserved, low-income, and racial/ethnic minority populations negatively affected by social determinants of health are less likely to access eye care and complete diabetic eye exams, yet are more likely to have DED and worse diabetes-related health outcomes (43–47). Implementation of accessible programs for diabetic eye exams using autonomous AI in underserved and under-resourced communities may improve early detection of DR, allow health systems to focus resources required to provide specialist eye care for patients with referable disease, and thereby improve visual outcomes and promote health equity in these populations.

We hope this review will serve as a practical guide, providing useful “nuts-and-bolts” and “how-to” implementation recommendations for anyone planning to deploy autonomous AI testing for DED in a real-world clinical setting. In addition to the practical considerations raised herein, there are research and ethical considerations regarding deep learning, machine learning, and big data. These issues, including data shift over time, bias, and health equity, are important but beyond the scope of this review.

This work was supported by the National Eye Institute (NEI) of the National Institutes of Health under award number R01EY033233 to R.M.W. R.C. receives research support in part from an unrestricted grant from Research to Prevent Blindness, Inc., to the University of Wisconsin–Madison Department of Ophthalmology and Visual Sciences, and from the NEI under award number 5K23EY030911-03. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

M.D.A. is an investor in, director of, and consultant to Digital Diagnostics, Inc.; has patents and patent applications assigned to the University of Iowa and Digital Diagnostics that are relevant to the subject matter of this article; is chair of the Healthcare AI Coalition; is a member of the American Academy of Ophthalmology AI Committee, the AI Workgroup Digital Medicine Payment Advisory Group, and the Collaborative Community for Ophthalmic Imaging (CCOI); and is chair of the Foundational Principles of AI CCOI Workgroup. No other potential conflicts of interest relevant to this article were reported.

R.M.W., R.C., M.D.A., and T.Y.A.L. wrote the manuscript. All of the authors critically revised the manuscript and read and approved the final manuscript for submission. R.M.W. is the guarantor of this work and, as such, has full access to all the data presented and takes responsibility for the integrity of the data and the accuracy of the review.