Performance of Artificial Intelligence in Detecting Diabetic Macular Edema From Fundus Photography and Optical Coherence Tomography Images: A Systematic Review and Meta-analysis

Diabetic retinopathy (DR) is a neurovascular complication of diabetes (1,2). Diabetic macular edema (DME), which can develop at any stage of DR, is the primary cause of irreversible vision loss in people with diabetes (1,3), overtaking proliferative DR as the most frequent cause of visual impairment among people with diabetes in developed countries (4). As blindness from DME is preventable with timely treatment, it is important to precisely identify DME among people with diabetes. Given that the population with diabetes is projected to approach 600 million worldwide by 2035 (5), DME is likely to be responsible for substantial vision loss unless detected earlier and treated adequately in the future.

DR screening programs are currently implemented at the primary care level in many countries using two-dimensional (2D) nonstereoscopic digital fundus photography (FP) using fundus cameras (6–9). When signs of DME or other sight-threatening DR are identified, patients are referred to ophthalmologists for further clinical examination and management. However, the diagnosis of DME requires identification of retinal thickening, which is a three-dimensional (3D) concept that is difficult to be reliably diagnosed based on 2D FP. Notably, in screening settings, manual interpretation of FP images for DME has been reported with high false-positive rates (e.g., >86% in Hong Kong [10] and >79% in the U.K. [11]), causing unnecessary referral of suspected DME to ophthalmologists and leading to a substantial increase in medical costs and waiting time for patients.

Spectral domain optical coherence tomography (SD-OCT) is a noninvasive imaging modality that provides 3D volumetric scans of the layered retinal structures. It has been widely used as the gold standard for DME diagnosis in clinical settings and clinical trials (12,13), monitoring treatment response, and providing prognostic information. Its role as a screening tool for DME has been investigated in pilot studies (13,14), showing that implementation of OCT into DR screening programs provides a reduction in referrals for diabetic maculopathy by 40% (15). Nevertheless, commercially available OCT devices could be at least three times more expensive than fundus cameras. Therefore, further studies are required to demonstrate the feasibility and cost effectiveness of implementing OCT into DME screening. More importantly, one common and critical issue for using FP or OCT for DR screening is the requirement of professionals to review a tremendous number of images. Taking OCT for example, a volumetric data cube usually comprises >100 images per eye. Manual slice-by-slice assessments are required to ensure that no positive cases are missed. Considering the large population of individuals with diabetes, this is indeed a time- and labor-intensive task.

Artificial intelligence (AI), particularly deep learning, is a major area of research in medical image analysis (16). Recently, deep learning has been making remarkable breakthroughs in the field of DR for revolutionizing DME detection from FP and OCT images in an automated, convenient, and efficient fashion (17,18). Despite the excellent diagnostic performance, several gaps remain. First, whether FP-based AI can obtain satisfactory performance for screening DME has not yet been evaluated comprehensively. Second, although OCT is regarded as the gold standard for DME diagnosis, whether implementing OCT-based AI in DR screening provides significantly better performance requires validation studies. Third, factors that determine the AI’s discriminative performance in detecting DME also require further elucidation.

We conducted a meta-analysis to evaluate the synthesized discriminative performance of AI in detecting DME from FP or OCT images and to identify factors that affect a model’s performance. In addition, on the basis of current literature, we further discuss potential research directions, aiming to facilitate clinical translation of AI models to real-world practice for DR screening.

We conducted a systematic review and meta-analysis to assess the diagnostic performance of AI for detecting DME from FP and OCT images in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines (19). Institutional review board approval and informed consent were not required, as all data were extracted from past publications. All research adhered to the principles of the Declaration of Helsinki.

Studies that used AI algorithms to detect DME from FP or OCT macular scans from 1 January 1991 to 12 February 2023 were included in the current study. We excluded records given the following reasons: 1) did not specify patients’ diabetes status; 2) did not provide qualitative outcome of DME (i.e., presence or absence of DME); 3) used traditional computer vision methods instead of AI techniques; 4) did not report sensitivity, specificity, and area under the receiver operating characteristic (AUROC) curve; 5) used ocular images other than FP or OCT macular scan as input; 6) were conference abstracts, ongoing studies, reviews, meta-analyses, comments, editorials, book chapters, theses, non–English-language, and nonhuman studies; and 7) did not have full text available. We also excluded any studies of low quality using the QUADAS-2 tool (20).

Two independent reviewers (C.L. and Y.L.W.) conducted the literature search in seven electronic libraries, including PubMed, Embase, Web of Science, Google Scholar, Scopus, the Cochrane Library, and CINAHL, using a hierarchical search strategy with combinations of keywords related to AI (e.g., artificial intelligence, AI, machine learning, deep learning, automated detection), the target condition (e.g., DME), and image modalities (e.g., FP, OCT) and Medical Subject Headings terms, as appropriate. Full details of the search strategy for each database are described in Supplementary Table 1.

We ran our search strategy through the electronic libraries and collected all search results using EndNote. Duplicates were removed before selection. The selection process was in three phases. First, two reviewers (C.L. and Y.L.W.) independently screened the titles and abstracts of the records and identified studies related to the topic. Second, the two reviewers (C.L. and Y.L.W.) independently assessed the full texts and excluded articles that met the exclusion criteria. Third, we evaluated the quality of included studies using the QUADAS-2 tool and eliminated studies of low quality. Each study was evaluated for risk of bias and applicability following four key domains: patient selection, index test, reference standard, and flow and timing. Reference lists of the selected studies were manually searched and screened. During the entire process, discrepancies between the two reviewers were resolved by discussions with a senior reviewer (Z.T.).

Relevant data were extracted from the included studies, including 1) study characteristics (author names, year of publication, and country), 2) data set characteristics (source of database, imaging modality and device, imaging protocol, image resolution, number of participants, eyes and images, number of participants and eyes with DME, number and experience of graders, and DME definition [i.e., FP, OCT, ophthalmoscopy]); 3) algorithm characteristics (types of networks; data splitting and data distribution in training, testing, and validation sets; and outcomes), and 4) performance metrics (AUROC; sensitivity, recall, or true-positive rate; specificity or true-negative rate; false-positive rate; accuracy; false-negative rate; precision or positive predictive value; negative predictive value; and other reported performance parameters).

We constructed a 2 × 2 contingency table for each study in RevMan (version 5.4.1) with the built-in calculator using the collected study characteristics and performance parameters. We imported the 2 × 2 contingency tables into R (version 4.2.0) and used the mada package (21) and VassarStats (22) to perform statistical analyses. To demonstrate the discriminative performance of FP- and OCT-based AI for DME detection, respectively, we calculated the pooled AUROC using a hierarchical model. Pooled sensitivity and specificity were calculated by bivariate random-effects model to measure the AI’s ability to identify DME-positive or DME-negative eyes. Bivariate analysis produces informative summary measures in diagnostic reviews. As a single indicator, the diagnostic odds ratio (DOR) is used to evaluate how much greater the odds of a positive diagnosis of DME is for patients with test-positive versus test-negative results, providing a comprehensible overview of the models’ overall diagnostic performance (23). We generated heterogeneity parameters, forest plots, and summary receiver operating characteristic (SROC) curves. Subgroup analyses were performed according to the type of AI (machine learning vs. deep learning), data set size (smaller than median vs. larger than median), data set diversity (single vs. multiple), and testing sets (internal vs. external). Small developmental data set size was defined as smaller than the median number of images among studies included in this meta-analysis, and multiple training data sets were defined as data from different institutions or with systematically different population characteristics (18). We also created funnel plots to assess for any publication bias in our meta-analysis.

The selection process for 53 included studies is shown in Fig. 1. Initially, 4,776 search results were identified, and 1,338 duplicates were removed before selection. We screened the titles and abstracts of the 3,438 records, and 2,796 were excluded because of irrelevant topics. Of 642 studies for full-text review, 589 were excluded according to the exclusion criteria. We assessed the quality of 53 studies, and none were excluded for low quality (Supplementary Fig. 1). Finally, 53 studies were included in the meta-analysis (full citations provided in the Supplementary Material).

Basic characteristics of the included studies are presented in Table 1. Among the 53 studies, 25 used FP as input images, while 28 used OCT B scans, among which 2 studies also included volumetric scans as input data for algorithm development. For FP studies, 8 developed machine learning algorithms, and 17 developed deep learning algorithms. For OCT studies, 3 studies developed machine learning algorithms, and 25 developed deep learning algorithms. Heidelberg Spectralis SD-OCT was most commonly adopted (21 studies), while Cirrus HD-OCT, Cirrus SD-OCT, Topcon Triton OCT, Topcon 1000 SD-OCT, Topcon 3D OCT-1 Maestro, and Optovue RTVue-XR Avanti were also adopted in other studies. Regarding publicly available data sets for training, testing, and validation, the Methods to Evaluate Segmentation and Indexing Techniques in the Field of Retinal Ophthalmology (MESSIDOR) and Kermany et al. (24) data sets were the most common for FP and OCT studies, respectively.

Detailed statistical results are shown in Table 2. FP-based AI of 25 studies yielded a pooled AUROC of 0.964 (95% CI 0.964–0.964) with a sensitivity of 92.6% (95% CI 90.2–94.4%), specificity of 91.1% (95% CI 88.0–93.4%), and DOR of 147.3 (95% CI 94.0–230.8), while OCT-based AI of 28 studies yielded a pooled AUROC of 0.985 (95% CI 0.985–0.985) with a sensitivity of 95.9% (95% CI 94.1–97.2%), specificity of 97.9% (95% CI 96.6–98.6%), and DOR of 1,154.6 (95% CI 691.0-1,929.1). The overall pooled AUROC, sensitivity, specificity, and DOR of the 53 studies was 0.979 (95% CI 0.979–0.979), 94.6% (95% CI 93.1–95.7%), 95.8% (95% CI 94.3–96.9%), and 456.1 (95% CI 311.9–667.1), respectively. Forest plots and SROC curves of FP and OCT studies are shown in Fig. 2A–D.

Regarding the type of AI, deep learning algorithms showed higher pooled AUROC (0.979) than machine learning (0.966, P < 0.001). Regarding the developmental data size, the median image number (8,655) was used to stratify algorithms into using smaller and larger data sets. For studies that proposed multiple algorithms or models that used separate developmental data sets, each model was evaluated separately. Results demonstrated that developmental data sets larger than the median (i.e., >8,655) were associated with better validation results with higher pooled AUROC, sensitivity, specificity, and DOR (0.981, 95.3%, 96.1%, 575.0, respectively) than those developed with data sets smaller than the median (0.975, 93.6%, 95.5%, 353.4, respectively).

Regarding testing data sets, 50 studies were internally validated, but only 13 studies performed external validation. Among externally validated studies, we compared their performances upon internal and external validation. Results demonstrated lower pooled AUROC, sensitivity, specificity, and DOR for the algorithms when validated externally (0.967, 91.7%, 93.4%, 1,089.5, respectively) than internally (0.985, 96.9%, 97.5%, 162.6, respectively). Moreover, upon external validation, models trained on multiple data sets showed slightly better performance than those trained on a single data set (pooled AUROC 0.971 and DOR 187.9 for multiple data sets and 0.965 and 158.3, respectively, for a single data set, P = 0.002).

As evidenced by Supplementary Fig. 2A and B, bias at the bottom-right from the asymmetrical funnel plot is shown, which could result from large DOR and SEs among certain studies. Studies with small SEs also distributed among a rather scattered range of DOR, which did not concentrate around the funnel. Therefore, the results should be interpreted with caution and indicate the need for further rigorous research to reduce bias.

The present meta-analysis investigated the performance of AI in detecting DME using FP or OCT images. Overall, our results indicate good discriminative performance for both FP- and OCT-based AI models in terms of pooled AUROC, sensitivity, specificity, and DOR. Potential factors that may increase model performance include the type of AI, sample size, and diversity in the developmental data set.

Among 25 FP-based AI studies, the pooled AUROC was 0.964, with sensitivity and specificity all >90%. This indicates the potential application of FP-based AI in primary care settings, considering the appreciable affordability and accessibility of FP. It may serve as an ideal tool to aid health care providers, especially in resource-restrained areas, to avoid intense human and time input. For example, as FP has already been used for DR screening, AI can be used as a first-pass tool and assist human graders’ subsequent grading in existing screening programs (6,7). This benefits the population with diabetes and the public health infrastructure by enhancing clinical workflow and referrals.

However, we noticed incomprehensive reporting in details of ground truth labeling for each study, i.e., whether it was based on FP alone or with additional clinical examinations (Supplementary Table 3). Since FP alone cannot serve as the gold standard for DME diagnosis, AI trained on these data may not overcome the previously reported high false-positive rate in current screening programs (10,11), despite better expertise (retinal specialists and ophthalmologists) for labeling. Thus, standardization of reporting in terms of the labeling gold standard and training materials would be recommended for future AI studies. Developing models with OCT-based diagnosis for the grading of its paired FP (25) may also improve the quality of ground truth labeling.

The 28 OCT-based AI studies that detect retinal thickening or DME features, such as intraretinal or subretinal fluids, yielded a pooled AUROC of 0.985, with sensitivity and specificity >95%. Previous publications attributed the improved performance to OCT providing 3D scans of the whole retinal structure, which are more informative for AI models to detect eyes with DME, especially among cases with central subfield thickness <300 μm (26). Practically, cases alike could be difficult to differentiate from normal ones by using 2D FP alone. The high discriminative performance obtained by OCT-based AI demonstrates its potential to be applied in tertiary settings and eye hospitals, especially in more resourced settings where OCT machines are affordable, for not only detecting the presence of DME but also further classifying center-involved DME (CI-DME) and non-CI-DME. This is of clinical value as eyes with CI-DME require urgent intervention, such as anti–vascular endothelial growth factor injection, while for eyes with non-CI-DME, initial observation is an acceptable option. Moreover, Tang et al. (17) developed deep learning models that used whole 3D volumetric scans to not only detect DME with classification of CI-DME or non-CI-DME but also identify non-DME retinal abnormalities, such as epiretinal membrane and macular hole. This implies another advantage of OCT-based AI models for efficiently differentiating different retinal abnormalities based on comprehensive information provided by the volumetric scans, which may facilitate more personalized treatment plans for patients.

On the other hand, analysis of 3D volumetric OCT images may be useful for future algorithm development. In our meta-analysis, only two studies (17,27) developed OCT-based algorithms that detected DME from 3D volumetric scans directly. As the data remain insufficient for subgroup analysis, we have yet to evaluate and compare the accuracy between using OCT 2D B scans and 3D volumetric scans. Previous studies suggested substantial merits of deep learning training with 3D volumetric scans in terms of labeling effort (28,29). Compared with volume scan–level annotation, ground truth labeling of B scans is more labor intensive and time consuming (17). Model output of one result for one volumetric scan may also facilitate clinical application for graders with less expertise. However, analysis of volumetric scans requires higher computation power. Therefore, future meta-analyses may compare the performance of DME detection by AI using OCT B scan and volumetric scan and evaluate whether using volumetric scans as input provides additional benefits in terms of model development and clinical application.

We also conducted subgroup analyses to investigate how the type of AI, developmental data set size, validation methods, and number of training data resources affect the performance of AI algorithms.

Regarding AI type, we found that both deep learning and traditional machine learning algorithms obtained satisfactory performance with pooled AUROC, sensitivity, and specificity >90%, while deep learning algorithms demonstrated a slightly higher pooled AUROC. While traditional machine learning requires complex feature engineering (29), most included studies adopted deep learning, which allows automatic pattern recognition with high-volume modeling capability and may offer identification of insignificant markers of early-stage DME. Since deep learning requires higher computation power and larger training data sets (30), some studies further used transfer learning and semisupervised learning techniques. Transfer learning allows pretrained networks to be retrained on specific tasks with fewer training examples and less computational power, while semisupervised learning, such as generative adversarial networks, addresses the scarcity of labeled data. Self-supervised learning, a novel technique combining qualities of high label efficiency and strong generalization capacity, indicates the great potential of deep learning techniques in accelerating medical AI development.

Our results show that developing AI with a larger data set provides better performance. This supports the outcomes of previous subsampling experiments, which showed that AUROC of an FP-based deep learning model for DME detection increases with sample size (25), and the performance of another FP-based deep learning model for referable DR increased with training data set size, plateauing at ∼60,000 images (18). However, we also noticed a discrepancy from individual studies. Ai et al. (30) developed two models for DME detection, a complete model trained on a large, but imbalanced data set of >60,000 normal and DME images and a limited model trained on a small, but balanced data set of 1,000 images only from each class. The limited model was shown to outperform the complete model despite a huge deficiency in number of training data. However, in our meta-analysis, we have yet to compare the effect of class balance and sample size on model development and performance.

Apart from data acquisition, we noticed deficiencies in external validation among studies included in our meta-analysis, which was performed in only 13 of 53 studies. External validation is important since overfitting is a common pitfall in AI training wherein models learn specific patterns in the training data and show strong performance in similar internal testing data, resulting in great discrepancies in performance between internal and external validation. This is supported by our subgroup analysis, where algorithms showed a significant decline in all pooled performance parameters when validated with external unseen data. Therefore, guidelines related to AI development and performance reporting emphasize the importance of external validation before clinical translation (31,32). Insufficiency in external validation among the included studies limits our assessment of the generalizability of algorithms in the real world and may lead to an overestimation of their performance in clinical settings. Future research should be directed toward external validation of algorithms.

Regarding training data diversity, we found better performance upon external validation among studies trained on data from multiple institutions or with systematically different population characteristics. Greater data diversity from multiple sources may enhance the resilience of algorithms over different baseline populations in unseen external data. This suggests that increasing developmental data diversity may enhance generalizability of AI upon clinical application to maintain its performance in detecting DME among patients with various baseline characteristics.

We also noticed discrepancies in baseline characteristics between validation data sets and real-world data. Epidemiological studies showed that DME prevalence is 1.4–12.8% among the population with type 2 diabetes worldwide (33,34). However, the proportion of DME images among validation data sets of included studies has a median of 27.5% (Supplementary Table 3), which is much higher than the general prevalence. Such discrepancy may hinder our understanding of the models’ realistic applicability in primary settings. Therefore, validation of AI using data sets with DME prevalence more resemblant of current epidemiology may provide more representative statistics. In view of the unrepresentative DME prevalence in retrospective data sets, it is also important to perform real-world prospective validation studies before these models can be used clinically with confidence.

Our meta-analysis has several strengths. First, our study demonstrated that machine learning and deep learning could be used to develop AI with high diagnostic performance for detecting and classifying DME from FP or OCT images. From a clinical and public health standpoint, these advancements can enhance clinical workflow and optimize identification of patients requiring specialists or even retina clinic referrals. FP-based AI could be incorporated into existing DR screening programs, while OCT-based AI could be used as a secondary tool to screen individuals with positive results based on FP and determine treatment options. Second, we identified factors that may enhance model performance, such as deep learning technique, a larger developmental data set, and greater training data diversity. Third, regarding gaps in existing research, we recognized insufficiencies in external validation and discrepancies between validation data and the realistic, primary care-level populations. Addressing these gaps may enhance clinical translatability of these AI models and accelerate their implementation in real-life settings.

Our study also has several limitations. First, the outcomes of algorithms were not standardized and were labeled based on different disease definitions. For example, we noticed that there are variable outcomes of algorithms, i.e., DME, clinically significant macular edema (CSME), nonclinically significant CSME (non-CSME), CI-DME, non-CI-DME, and referable DME, of which CSME and non-CSME were graded based on the exudate location on FP and retinal thickness or DME features on OCT, while CI-DME and non-CI-DME are graded based on retinal thickness on OCT. There is also no standard definition of referable DME. Clinically, this may cause difficulty in results interpretation and, thus, referral procedure. Future guidelines should standardize model outcome and disease definition to facilitate clinical translation and allow fair performance evaluation. Second, there are insufficient data for evaluating model performance on comparing OCT 2D B scans and 3D volumetric scans, OCT machines, and generalizability of models on external data sets. Third, the small sample size of some included studies may reduce representativeness of their AI performance. Fourth, few articles reported detailed demographics of their database, such as age, sex, and duration of diabetes, which provided insufficient data for meta-regression to evaluate their database diversity and their effect on AI development. Finally, there is also a vacuum for assessing whether AI performance might be affected by different DME severity. Future research based on DME grade is required to further elucidate the performance discrepancies in AI algorithms.

For future development of AI models, data acquisition remains a major challenge in terms of data diversity and sample size. Collaboration between multiple centers or integration of multiple public data sets with different baseline characteristics would be beneficial in increasing the diversity and size of developmental data. Data efficient techniques, such as generative adversarial networks (35), synthetic minority oversampling technique (36), few-shot transfer learning (37), and self-supervised learning, may aid algorithm development among a limited-source training sample. However, to date, there are no studies that have provided an estimation of optimal sample size and class balance for DME model development. Further studies on sample size determination methodologies are required to estimate the developmental sample size that balances diagnostic performance with resource availability (38). Standardization on reporting of labeling gold standard, patient demographics, such as age, sex, ethnicity, type and duration of diabetes, and settings of image acquisition, would also facilitate the assessment of the value of AI as a potential population screening tool for DME. Clinically, AI-relevant education of health care providers and development of a systematic approach for AI implementation by multiple stakeholders are crucial, including legislation addressing issues of accountability and data sharing (39).

In conclusion, AI algorithms show satisfactory discriminative performance in DME detection from either FP or OCT images. Potential factors that may increase model performance include the type of AI, sample size, and diversity in the developmental data set. There remain significant gaps in external validation among current studies to evaluate models’ generalizability for clinical translation. Further studies can evaluate the potential and added value of OCT volumetric scan analysis by AI models, estimate the optimal sample size and effects of class balance and patient demographics, and compare the relative effectiveness of AI in DME detection with human evaluation.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. C.L. and Y.L.W. researched data; performed the systematic search, study selection, data extraction, and statistical analyses; and wrote the manuscript. Z.T. and A.R.R. researched data, contributed to the discussion, and reviewed and edited the manuscript. X.H., T.X.N., D.Y., and S.Z. conducted the data extraction and reviewed and edited the manuscript. J.D., S.K.H.S., and C.Y.C. contributed to the discussion and reviewed and edited the manuscript. All authors approved the decision to submit for publication. A.R.R. and C.Y.C. are the guarantors of this work and, as such, had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.