A Machine Learning–Based Predictive Model to Identify Patients Who Failed to Attend a Follow-up Visit for Diabetes Care After Recommendations From a National Screening Program

Diabetes poses a burden on health care and the economic, social, and communal lives of patients, and patients need community-based support to live normally (1). Medical intervention immediately after the diagnosis of diabetes is important to achieve a good patient prognosis (2). However, individuals who are suspected of having diabetes at medical checkups frequently fail to attend follow-up visits for diabetes care. Indeed, in Japan, 65% of patients with a positive fasting blood glucose test or glycated hemoglobin (HbA_1c) ≥6.5% (HbA_1c ≥48 mmol/mol) failed to attend a follow-up visit for diabetes care (3). In addition, as reported in the U.K., 30% of patients invited for screening failed to undergo the first screening test, and 10% failed to undergo the second screening test (4).

Although factors for discontinuation after the initiation of diabetes care have been studied (5), little is known about the factors related to failure to attend a follow-up medical visit for newly screened diabetes after a checkup. A previous study using data from 2005 to 2010 with a moderate sample size (n = 3,878) showed that younger age and lower blood glucose levels were associated with the failure to attend a follow-up visit in patients with newly screened hyperglycemia (3). Because the diagnostic criteria for diabetes changed in 2010 (6), investigating the factors related to failure to attend a follow-up visit based on the recommendations at checkups is important to develop an optimal strategy for patients with newly screened diabetes. Accumulated evidence has emphasized the importance of strict interventions in patients with diabetes (7). Therefore, the development of a predictive model for failure to attend a follow-up visit for newly screened diabetes at checkups should attract the attention of policymakers and payers because of the expectation that interventions targeting this population may be important in reducing the diabetes-related health care burden.

In this context, we aimed to explore the factors and develop a machine-learning model for predicting the failure to attend a follow-up visit among patients who were positive for diabetes during screening at annual checkups using a claims database, containing data for laboratory values, and a lifestyle questionnaire, after checkups piloted by the Japanese government for prevention of lifestyle-related diseases.

This was a retrospective cohort study using a commercially available database, the JMDC claims database (Tokyo, Japan), of patients whose details have been previously described (8). Briefly, the JMDC database contains claims data of ∼10 million insured individuals as of 2020, most of whom are company employees and their families. Diagnoses were recorded using the ICD-10, and drug specifications were based on World Health Organization Anatomical Therapeutic Chemical Classification System (WHO-ATC) codes.

The database additionally contains information on annual checkups for lifestyle-related diseases (9), which covers 4.7 million insured individuals. Employers have been legally required to provide full-time employees with annual checkups, and the Japanese government initiated universal lifestyle-related disease screening throughout Japan in 2008, which focuses on metabolic syndromes to prevent lifestyle-related diseases. With this scheme, insurers are obliged to provide universal lifestyle-related disease screening. When individuals are found to have abnormalities in their checkups at the time of screening, they are encouraged to receive advice from a public health nurse or consult a physician for disease management. Individuals in a hyperglycemic state during a checkup, where the threshold is set as fasting blood glucose ≥126 mg/dL or HbA_1c ≥6.5% (≥48 mmol/mol), are advised to consult a doctor (9). The JMDC database contains the following results of the checkups: blood pressure recordings, clinical laboratory tests, such as complete blood count, blood glucose, biochemistry, and urinary dipstick test, as well as questionnaires on lifestyle factors, such as smoking and alcohol habits.

Based on previous studies (3,4), we included individuals who met all of the following criteria: 1) age ≥40 years, 2) fasting blood glucose ≥126 mg/dL and HbA_1c ≥6.5% (HbA_1c ≥48 mmol/mol) at the baseline checkup from April 2007 to October 2019 without any missing values among all of these variables, and the combination of the two glycemic marker levels satisfied the diagnostic criteria for diabetes (10). For each patient, we set the baseline period as the time between 1 year prior to the checkup and the day before the checkup. During the year prior to the checkup, it was imperative that there were no diabetes-related claims, as described previously (3). Therefore, all included individuals were considered to be “patients with newly diagnosed diabetes.” Follow-up visits for diabetes were defined as having any claims related to diabetes, including HbA_1c or glycated albumin tests and prescriptions for diabetes. We excluded individuals who were censored; in other words, those who did not receive insurance coverage throughout the observation period.

The study protocol was approved by the Institutional Review Board of the Graduate School of Medicine of The University of Tokyo. The requirement for informed consent was waived due to the anonymous nature of the data.

The following information was obtained from the database: sex, age, BMI, waist circumference, blood pressure, serum γ-glutamyl transferase, creatinine, uric acid levels, HbA_1c levels, and urinary protein/glucosuria dipstick results. We also obtained results from the checkup of the previous year for glycemic markers, such as fasting blood glucose or HbA_1c levels. We classified BMI into four categories as follows: underweight (<18.5 kg/m²), normal weight (18.5–24.9 kg/m²), overweight (25.0–29.9 kg/m²), and obesity (≥30.0 kg/m²). Binary variables were used to determine the medication prescriptions for dyslipidemia (WHO-ATC codes starting with C10), hypertension (WHO-ATC codes starting with C02, C03, C04, C07, C08, or C09), hyperuricemia (WHO-ATC codes starting with M04), and depression (WHO-ATC codes starting with N06A or N06BA), from the claims data during the previous 12 months. We also set a binary variable indicating whether the individual was an insured employee. Blood pressure was classified into four categories based on the definition of the Japanese Society of Hypertension: normal or grade 1–3 in hypertension (11). We also obtained responses of questionnaires on lifestyle factors (smoking and alcohol habits), medical history of stroke or ischemic heart disease, health behavior changes based on the transtheoretical model, and willingness to receive health instruction from a public health nurse.

Candidate risk factors were selected based on previous studies (3,4) and clinically relevant variables. Specifically, the candidate risk factors included patient age; sex; BMI; smoking habits or alcohol intake; prescription histories of antihypertensives, antidyslipidemic, antihyperuricemic, or antidepressive agents in the previous 12 months; glycosuria (negative or positive) or urinary protein (negative or trace/+1 or worse) results from a urinary dipstick test; hemoglobin values; and whether the patient was insured or a dependent (3,4). We also included the following variables that were clinically important in a previous study that examined the discontinuation of follow-up visits in patients with diabetes: the number of months when an individual used a medical facility in the past 12 months prior to the checkup, health behavior changes based on the transtheoretical model, and willingness to receive health instruction from a public health nurse (12). In addition, we added the following items related to severity or comorbidity: estimated glomerular filtration rate, fatty liver index (13), and whether a patient was recommended to consult a doctor for hyperglycemia (fasting blood glucose ≥126 mg/dL or HbA_1c ≥6.5% [≥48 mmol/mol]) in the previous year.

The primary outcome was the failure to attend a follow-up visit based on recommendations at the checkups, which was defined as a person failing to attend a follow-up visit within 6 months after undergoing a checkup in which the laboratory data satisfied the diagnosis of diabetes, despite being recommended to consult a physician. We defined a visit to a physician during the 6 months by recorded consultation with a physician with a registered name of diabetes (ICD-10 codes: E10, E11, E12, E13, or E14), existence of measurement of glycemic markers, such as HbA_1c or glycated albumin, or prescription of antihyperglycemics.

We summarized the background characteristics of the eligible patients according to the failure to attend a follow-up visit. Categorical variables were compared using the χ² test, and continuous variables were compared using the Wilcoxon rank-sum test.

We classified the study patients into a training set consisting of randomly selected 80% of the total participants and a test set comprising the remaining participants (20%).

In the training set, we developed a reference model and a machine-learning model to predict the probability of failing to attend a follow-up visit. For the reference model, we fitted a previously reported logistic regression model (without penalization), including all 13 candidate predictors used in the previous study (3). We calculated odds ratios (ORs) and 95% CIs for failure to attend a follow-up visit using all of these variables, in which the BMI was treated as a continuous variable. For the machine-learning model, we fitted a logistic regression with Lasso regularization (Lasso regression) among all of the candidate variables (14). Lasso regression is an efficient model that both shrinks and selects regression coefficients (14,15) and allows for better interpretability of the model. The optimal value for the hyperparameter λ was selected to maximize the percentage of correctly identified case/control subjects with 10-fold cross-validation on the training set before using the remaining 20% of the data to test the predictability of the model. We applied the maximal λ within 1 SE of the difference. Lasso regression with the 1-SE rule can yield a more parsimonious model than the model using the minimal λ (16,17). Based on the Lasso regression model, we identified factors related to failure to attend a follow-up visit. In addition, we conducted a logistic regression model using the identified factors for postestimation (i.e., to calculate the OR and 95% CI of each factor for the failure to attend a follow-up visit).

In the test set (the remaining 20% of the sample), we measured the predictive performance using C-statistics (area under the receiver operating characteristic [ROC] curve). The C-statistics between the models were compared using the Delong test. We also obtained the values of the Bayesian information criteria for each model.

We performed an analysis stratified by the severity of diabetes because the severity may affect the intensity of failure to attend a follow-up visit, as in a previous study (3). We set a cutoff value of HbA_1c of 8.0% (≥64 mmol/mol) because this value is a less stringent target for most patients with diabetes, as defined in the guidelines (10,18). We also performed another analysis stratified by whether the patient received antihypertensive, antidyslipidemic, or antihyperuricemic agents in the previous 12 months.

As sensitivity analyses, we also explored factors related to the failure to attend a follow-up visit based on recommendations in checkups within 3 and 9 months after the checkup.

Statistical significance was set at P < 0.05. All analyses were performed using Stata version 17 software (StataCorp, College Station, TX).

Among the patients whose HbA_1c and fasting blood glucose levels were available from checkups from April 2007 to October 2019, we identified 11,023 patients who tested positive for diabetes at the checkups but had no history of treatment or diagnosis in the previous year. We subsequently excluded 378 patients who were censored during the first 6 months after the checkups, and the remaining 10,645 patients were eligible for the analysis.

Overall, 5,450 (51.1%) patients failed to attend a follow-up visit based on recommendations at the checkups within 6 months (Table 1). Briefly, those who failed to attend a follow-up visit for newly screened diabetes were likely to be male, younger, smokers, frequent drinkers, and insured people. The results of their checkup showed a higher proportion of people with negative results for glucosuria or urinary protein and lower HbA_1c values. In addition, they were less likely to have a prescription history of antihypertensive, antidyslipidemic, antihyperuricemic, or antidepressive agents and to be frequent visitors of medical facilities in the previous 12 months.

The process of variable selection using the hyperparameter λ and the corresponding values of the mean squared error with each SE bar is shown in Fig. 1A. We first obtained λ_CV, with which the cross-validation function was minimized. Subsequently, we obtained λ_SE, the largest λ within 1 SE of the lowest mean squared error. Using λ_SE, we constructed the Lasso regression model for the failure to attend a follow-up visit, and four variables were selected (Fig. 1B). The most important factor for the failure to attend a follow-up visit for diabetes care was a lower frequency of physician visits in the previous year (OR 1.15/1 time decrease/12 months [95% CI 1.13–1.16]), followed by lower HbA_1c levels (OR 1.24/1% [11 mmol/mol] decrease [95% CI 1.20–1.27]), negative history of receiving antidyslipidemic prescription (OR 1.86 [95% CI 1.58–2.19]), and negative history of receiving antihypertensive prescription (OR 1.37 [95% CI 1.19–1.57]) (Fig. 2). Based on the previously reported logistic regression model, we used 13 variables, and the coefficients of the multivariable regression are shown in Supplementary Table 1.

The Lasso regression model using four predictors had a better discrimination ability than the previously reported logistic regression model using 13 predictors (C-statistic: 0.71 [95% CI 0.69–0.73] vs. 0.67 [95% CI 0.65–0.69], respectively; P < 0.001) (Fig. 3).

When stratified by HbA_1c values (HbA_1c <8.0% and ≥8.0% [<64 mmol/mol and ≥64 mmol/mol]), the results were similar to those of the primary analysis across the groups. Among those with HbA_1c <8.0% [<64 mmol/mol] (N = 7,485), 4,035 (53.9%) failed to attend a follow-up visit. Among those with HbA_1c ≥8.0% (≥64 mmol/mol) (N = 3,160), 1,415 patients (44.8%) failed to attend a follow-up visit. Additionally, the stratified analyses yielded a similar pattern of variable importance in the Lasso regression model (Supplementary Fig. 1). Although the Lasso regression model identified three factors in patients with HbA_1c of 6.5–7.9% (48–63 mmol/mol), four factors were identified in patients with HbA_1c ≥8.0% (≥64 mmol/mol). While HbA_1c levels were selected as an important variable in patients with HbA_1c ≥8.0% (≥64 mmol/mol), they were not in patients with HbA_1c <8.0% (<64 mmol/mol). The three common variables in both stratified groups were a lower frequency of physician visits in the previous year, negative history of receiving an antihypertensive prescription, and negative history of receiving an antidyslipidemic prescription. Supplementary Fig. 2 shows the ROC curves for each group, and the results were similar to those of the main analysis.

Among those receiving antihypertensive, antidyslipidemic, or antihyperuricemic agents (N = 2,896), 871 patients (30.1%) failed to attend a follow-up visit. Among those who did not receive any of these drugs (N = 7,749), 4,579 (59.1%) failed to attend a follow-up visit. These stratified analyses yielded a lower ability to predict failure (Supplementary Fig. 3). Among those receiving any of these drugs, the Lasso regression model using one variable (frequency of physician visits in the previous year) had a comparable discrimination ability with the previously reported logistic regression model using the 13 predictors (C-statistic: 0.60 [95% CI 0.55–0.65] vs. 0.61 [95% CI 0.57–0.66], respectively; P = 0.71). Among those who did not receive these drugs, the Lasso regression model using two variables (frequency of physician visits in the previous year and HbA_1c values) had a better discrimination ability than the previously reported logistic regression model using the 13 predictors (C-statistic: 0.66 [95% CI 0.63–0.68] vs. 0.61 [95% CI 0.58–0.64]; P < 0.001).

Supplementary Figure 4 shows the importance of variables in the failure to attend a follow-up visit during observation periods of 3 or 9 months after the checkup. In the model with observation periods of 3 months and 9 months, 64.3% and 45.0% failed to attend a follow-up visit, respectively. In both settings, the Lasso regression model involved the following four variables: lower frequency of physician visits in the previous year, followed by lower HbA_1c levels, negative history of receiving antidyslipidemic prescription, and negative history of receiving an antihypertensive prescription. The results of the ROC curves comparing the predictive ability of the developed models in both settings were also comparable to those of the main analysis (Supplementary Fig. 5).

In this analysis of 10,645 patients with newly diagnosed diabetes in an administrative database, the Lasso regression model using only four variables had a better discrimination ability to predict failure to attend a follow-up visit based on recommendations from a screening program, compared with the previously reported logistic regression model that used 13 variables. The important factors associated with the failure to attend a follow-up visit were lower frequency of physician visits in the previous year, followed by lower HbA_1c levels, negative history of receiving an antidyslipidemic prescription, and negative history of receiving an antihypertensive prescription.

A lower frequency of physician visits in the previous year and negative history of receiving antihypertensive or antidyslipidemic agents were associated with the failure to attend a follow-up visit. For one reason, patients treated for some diseases, especially diseases necessitating frequent follow-ups, are likely to have more opportunities to undergo an intervention based on the results of the checkups. Furthermore, patients treated for hypertension or dyslipidemia may be aware that it is necessary to treat lifestyle-related diseases. Otherwise, their primary physicians may help them understand the necessity. Another reason is that physicians who prescribe antidyslipidemic medication may be primary care physicians, diabetologists, or endocrinologists, who are specialized in diabetes care.

An implication of this study is that people who infrequently or rarely use medical facilities are at a high risk of not initiating diabetes care. This was confirmed by the stratified analyses, because all of the models failed to have a good predictive ability in those receiving antihypertensive, antidyslipidemic, or antihyperuricemic agents. To prevent failure to have a follow-up visit, whether the screened people used recent medical services should be considered. Patients with newly screened diabetes and no history of recent medical facility visits should be nudged to undergo a follow-up visit. Although nudging has been recognized as important (19), no studies have examined effective methods of nudging such patients; however, current evidence suggests that a multicomponent approach is promising (19). We should also consider supporting or giving incentives to employers who encourage or nudge their employees with newly screened diabetes to consult a physician to improve public health and social welfare.

We found a high proportion of patients who failed to attend a follow-up visit for diabetes care after checkup (53.2%), which was smaller than that in a previous study (65.2%) (3). The differences may be attributable to the different inclusion criteria between studies or the difference in recognition of the annual checkup system. While the previous study included patients with either elevated fasting blood glucose or elevated HbA_1c values, our study included patients who had both elevated fasting blood glucose and elevated HbA_1c values, which are diagnostic criteria for diabetes. Given that the severity of diabetes affects the initiation of diabetes care (20), it is understandable that patients satisfying both were likely to consult a doctor.

Our results should be interpreted in the context of the public health and health care system of Japan. The American Diabetes Association recommends community screening only after establishment of an appropriate referral system before testing (21), and the Japanese legally obliged checkup system complies with this requirement; the system provides employees with consultation to public health nurses or physicians based on checkup results (3). However, considering that nearly half of those recommended to see a physician failed to attend a follow-up visit in this study, some measures are warranted to improve the current situation. For example, in addition to nudging people with newly screened diabetes who rarely use a medical facility, it is necessary to consider implementing advocacy-oriented policy against diabetes in workplaces or among employers.

Our study has certain strengths and implications. First, we provided clinical information on a nationwide level predicting failure to attend a follow-up visit immediately after the diagnosis of diabetes, which is a clinically and socially challenging problem to be resolved. Second, our results could facilitate the implementation of community-based policymaking since the policymakers can focus on a smaller number of factors. Third, focusing on the limited number of variables was made possible with the use of Lasso regression with the 1-SE rule. At the same time, we succeeded in maintaining the predictive ability to detect the patients failing to attend a follow-up visit for diabetes care using a small number of variables while maintaining the discrimination capabilities using conventional logistic regression using 13 variables. Thus, we found a useful application of this parsimonious method in clinical decision-making.

We acknowledge four potential limitations of this study. First, while we selected patients with newly diagnosed diabetes by excluding patients who had received any diabetes-related care in the previous 12 months, some of the included patients might have been diagnosed with diabetes before. Second, although we identified 39 candidate factors to be included in the models, there may also be other factors affecting the failure to attend a follow-up visit. For instance, we did not obtain information on education level, occupation, and socioeconomic status, which was not recorded in the database. The third limitation may be that the effectiveness of screening for diabetes in terms of short-term health outcomes is not obvious (22). Lastly, our findings may not be generalizable to different settings. With a different medical insurance system or different cultural backgrounds, the predictive model in our study may not be completely applicable to patients urged to consult a physician for diabetes care. However, the variables used in this study are common in other countries, and the predictive model is, therefore, somewhat functional. A previous study in the U.K. showed that lower blood glucose levels and negative prescription histories were predictors of failure to attend a follow-up visit as in our study (4).

In conclusion, this retrospective cohort study using a large-scale administrative claims database revealed that a lower frequency of physician visits in the previous year, negative history of receiving antihypertensive or antidyslipidemic agents, and lower HbA_1c levels are important factors for failure to attend a follow-up visit for diabetes care. In particular, the information on medical facility utilization is important in predicting failure. The predictive model using only four factors may be beneficial for predicting failure to attend a follow-up visit for diabetes care. This finding may be useful in policymaking against lifestyle-related diseases.

Funding. This work was supported by grants from the Ministry of Health, Labour and Welfare, Japan (21AA2007) and the Ministry of Education, Culture, Sports, Science and Technology, Japan (20K18957, 20H03907, and 21H03159). This work was also supported by a junior scientist development grant from the Japan Diabetes Society to A.O.

Duality of Interest. A.O., S.Y., K.I.K., and T.K. are members of the Department of Prevention of Diabetes and Lifestyle-Related Diseases, which is a cooperative program between the University of Tokyo and Asahi Mutual Life Insurance Company. K.I.K. was employed by the Asahi Mutual Life Insurance Company. T.G. is employed by TXP Medical Co., Ltd. S.O. is a member of the Department of Eat-loss Medicine, which is a cooperative program between the University of Tokyo and ITO EN Ltd. No other potential conflicts of interest relevant to this article were reported.

Author Contributions. A.O., Y.H., T.G., S.Y., H.Y., and T.K. conceived and designed the study, performed the statistical analysis, and wrote, edited, and reviewed the manuscript. S.O., K.I.K., M.N., and T.Y. contributed to the discussion and interpretation of the data and edited and reviewed the manuscript. All authors have approved the final manuscript for publication. T.K. is the guarantor of this work and, as such, had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.