Recently, in Diabetes Care, Bogner et al. (1) reported results from the Prevention of Suicide in Primary Care Elderly: Collaborative Trial (PROSPECT) study that examined a care-management intervention for older primary care patients with depression. The authors reported that, compared with usual care, the intervention reduced 5-year all-cause mortality among diabetic patients with major or minor depression but not among depressed patients without diabetes. Although not mentioned by Bogner et al., an article by Gallo et al. in the Annals of Internal Medicine (2) that included all the authors of the Bogner et al. study reported that this same care-management intervention reduced 5-year all-cause mortality compared with usual care in the same cohort of patients with major depression but not in those with either minor depression or no depression. Gallo et al. noted that the effect of the intervention in patients with major depression “seemed to be limited to deaths due to cancer.”
Bogner et al. indicated that they tested the a priori hypothesis that depressed older adults with diabetes in practices randomized to the intervention would have lower 5-year mortality than depressed older adults with diabetes in usual care. It seems odd that the authors would test this hypothesis in the PROSPECT database, as their prior analysis of the data demonstrated an effect limited to cancer deaths (2) and the PROSPECT care intervention was not specifically designed to improve survival.
In a letter that recently appeared in the Annals of Internal Medicine (3), we expressed our concern that the statistical methods used by Gallo et al. resulted in model overfitting. As Babyak notes (4), “Overfitting yields overly optimistic model results: ‘findings’ that appear in an overfitted model don't really exist in the population and hence will not replicate.” In our letter, we noted that the automated variable selection methods the authors used for covariate selection “capitalize on variability unique to a given sample, radically underestimate the degrees of freedom used to determine estimates in regression models, often generate substantially inflated Type I error rates and artifactually small P values, and don't consistently produce replicable findings.” Bogner et al. used similar methods in their report on depressed patients with diabetes, although they switched from the P < 0.05 variable entry criterion used in Gallo et al. to a P < 0.10 entry criterion and included a somewhat different group of covariates. Both articles reported that the intervention did not have a significant effect on mortality before adjustment for covariates, yet both articles reported significant adjusted mortality effects.
Findings that suggest that important health outcomes like mortality can be affected by relatively simple psychosocial interventions are appealing. However, the conclusions of Bogner et al. are not scientifically justified and may raise false hopes and induce patients to seek psychosocial intervention with the goal of extending their lives. As Coyne et al. (5) pointed out in addressing similar issues in cancer research, unsubstantiated claims about survival benefits based on medical interventions are no less objectionable than similar unsubstantiated claims about herbal or coffee enemas as cures for cancer.
