0
Communications to the Editor |

Reporting of Predictive Logistic Models Should Be Based on Evidence-Based Guidelines FREE TO VIEW

Adrián V. Hernández; Yvonne Vergouwe; Ewout W. Steyerberg
Author and Funding Information

Affiliations: Erasmus Medical Center Rotterdam, the Netherlands,  Emory University School of Medicine Atlanta, GA

Correspondence to: Adrián V. Hernández, MD, MSc, Erasmus MC, Department of Public Health, Room Ee2010, PO Box 1738, 3000 DR Rotterdam, the Netherlands; e-mail: a.hernandez@erasmusmc.nl



Chest. 2003;124(5):2034-2035. doi:10.1378/chest.124.5.2034
Text Size: A A A
Published online

To the Editor:

Moss et al (March 2003)1 reviewed the deficiencies in reporting multivariable logistic regression analysis in the pulmonary and critical care medicine literature. They also suggested some potential guidelines to improve this reporting in both descriptive and predictive modeling. We would like to make some further suggestions, with emphasis on predictive models.

Recently, guidelines have been developed to adequately report randomized controlled trials (Consolidated Standards of Reporting Trials2 ), meta-analyses of randomized controlled trials (Quality of Reporting of Meta-analyses3 ), and diagnostic research studies (Standards for Reporting of Diagnostic Accuracy4 ). These guidelines were based on empirical evidence on factors affecting the reader’s understanding, validity, reliability, and generalizability of the findings. Even though predictive research studies are common in the literature, published guidelines are not sufficiently supported by such empirical evidence.56

Adequate reporting of predictive research should be based on elements that reflect how valid and precise the analyses were done. In predictive studies, overfitting is the key problem, which is related to several aspects of the modeling process. We suggest to report the number of candidate variables in addition to the variables in the final logistic model. The risk of overfitting after extensive modeling using many variables is high, especially in small data sets,78 and this unfortunately cannot be remedied by standard stepwise selection techniques.5 Also, a description of the choices underlying coding of variables and, in particular, selection of variables are of paramount importance.8 The number of outcome events must also be reported additionally to the number of total observations, because further overfitting is likely if the number of events per candidate variable is low, eg, < 10.,58 Attempts of internal validation (eg, cross-validation or bootstrapping) can also reduce overfitting, using statistical “shrinkage” of coefficients.,2,8 Further, predictive performance (calibration and discrimination) and internal/external validation should be described.7

We further suggest to avoid the reporting of some technicalities. The report of coefficients, SEs, and p values are not important, since the relevant information can be obtained from the odds ratios and their 95% confidence intervals of the final model variables.7 Moreover, we would not stress the report of collinearity in a predictive model, because we are primarily interested in the predictive performance of the whole model, but not in the regression coefficients of individual variables.78 If two variables are strongly correlated, no additional predictive information comes available once one is included in a predictive model.8 Even though we agree that collinearity is important in descriptive modeling, its report does not improve the judgement of the reader.

Predictive modeling using logistic regression analyses is becoming more important in the medical literature. Although Moss et al1 made an important contribution by noting where some deficiencies in reporting are, evidence-based recommendations for a proper reporting are still lacking and are urgently needed.

References

Moss, M, Wellman, DA, Cotsonis, GA (2003) An appraisal of multivariate logistic models in pulmonary and critical care literature.Chest123,923-928. [CrossRef] [PubMed]
 
Moher, D, Schulz, KF, Altman, DG The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials.Lancet2001;357,1191-1194. [CrossRef] [PubMed]
 
Moher, D, Cook, DJ, Eastwood, S, et al Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement.Lancet1999;354,1896-1900. [CrossRef] [PubMed]
 
Bossuyt, PM, Reitsma, JB, Bruns, DE, et al Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.BMJ2003;326,41-44. [CrossRef] [PubMed]
 
Harrell, FE, Jr, Lee, KL, Mark, DB Multivariate prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med1996;15,361-367. [CrossRef] [PubMed]
 
Laupacis, A, Sekar, N, Stiell, IG Clinical prediction rules: a review and suggested modifications of methodological standards.JAMA1997;277,488-494. [CrossRef] [PubMed]
 
Katz, MH Multivariate analysis: a primer for readers of medical research.Ann Intern Med2003;138,644-650. [PubMed]
 
Steyerberg, EW, Eijkemans, MJC, Harrell, FE, Jr, et al Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.Med Decis Making2001;21,45-56. [CrossRef] [PubMed]
 
To the Editor:

Dr. Hernandez and colleagues recommend additional considerations when reporting multivariable logistic regression analyses for predictive models. In our review of the pulmonary and critical care literature, only 6% of articles used multivariable logistic regression modeling in a predictive manner.1 Therefore, we focused our suggested requirements for the proper reporting of descriptive models that identify the effect of an individual variable on a specific outcome while adjusting for differences in other factors. In regard to predictive modeling techniques, we agree that reporting collinearity is less important. Like Hernandez and colleagues, we would encourage those interested in a more complete understanding of methodological standards for predictive modeling strategies to read the articles by Laupacis and colleagues2 in JAMA, and Wasson and colleagues3 in the New England Journal of Medicine.

References
Moss, M, Wellman, DA, Cotsonis, GA An appraisal of multivariable logistic models in the pulmonary and critical care literature.Chest2003;123,923-928. [CrossRef] [PubMed]
 
Laupacis, A, Sekar, N, Stiell, IG Clinical predictions rules: a review and suggested modifications of methodological standards.JAMA1997;277,488-494. [CrossRef] [PubMed]
 
Wasson, JH, Sox, HC, Neff, R, et al Clinical prediction rules: applications and methodological standards.N Engl J Med1985;313,793-799. [CrossRef] [PubMed]
 

Figures

Tables

References

Moss, M, Wellman, DA, Cotsonis, GA (2003) An appraisal of multivariate logistic models in pulmonary and critical care literature.Chest123,923-928. [CrossRef] [PubMed]
 
Moher, D, Schulz, KF, Altman, DG The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials.Lancet2001;357,1191-1194. [CrossRef] [PubMed]
 
Moher, D, Cook, DJ, Eastwood, S, et al Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement.Lancet1999;354,1896-1900. [CrossRef] [PubMed]
 
Bossuyt, PM, Reitsma, JB, Bruns, DE, et al Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.BMJ2003;326,41-44. [CrossRef] [PubMed]
 
Harrell, FE, Jr, Lee, KL, Mark, DB Multivariate prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med1996;15,361-367. [CrossRef] [PubMed]
 
Laupacis, A, Sekar, N, Stiell, IG Clinical prediction rules: a review and suggested modifications of methodological standards.JAMA1997;277,488-494. [CrossRef] [PubMed]
 
Katz, MH Multivariate analysis: a primer for readers of medical research.Ann Intern Med2003;138,644-650. [PubMed]
 
Steyerberg, EW, Eijkemans, MJC, Harrell, FE, Jr, et al Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets.Med Decis Making2001;21,45-56. [CrossRef] [PubMed]
 
Moss, M, Wellman, DA, Cotsonis, GA An appraisal of multivariable logistic models in the pulmonary and critical care literature.Chest2003;123,923-928. [CrossRef] [PubMed]
 
Laupacis, A, Sekar, N, Stiell, IG Clinical predictions rules: a review and suggested modifications of methodological standards.JAMA1997;277,488-494. [CrossRef] [PubMed]
 
Wasson, JH, Sox, HC, Neff, R, et al Clinical prediction rules: applications and methodological standards.N Engl J Med1985;313,793-799. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543