0
Correspondence |

ResponseResponse FREE TO VIEW

Mark T. Keegan, MB; Ognjen Gajic, MD, FCCP; Bekele Afessa, MD, FCCP
Author and Funding Information

From the Division of Critical Care, Department of Anesthesiology (Dr Keegan), the Division of Pulmonary and Critical Care, Department of Medicine (Drs Gajic and Afessa), and the Multidisciplinary Epidemiologic and Translational Research in Intensive Care (METRIC) group (Drs Keegan, Gajic, and Afessa), Mayo Clinic.

Address for correspondence: Mark T. Keegan, MB, Mayo Clinic, Department of Anesthesiology, Charlton 1145, 200 First St SW, Rochester, MN 55905; e-mail: keegan.mark@mayo.edu


Financial/nonfinancial disclosures: The authors have reported to CHEST that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article.

Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details.


Chest. 2013;143(3):876. doi:10.1378/chest.12-2783
Text Size: A A A
Published online
To the Editor:

We thank Drs Nathanson and Higgins for their interest in our work and recognize their contributions to the field through their development of the Mortality Probability Model (MPM) III. As they note, in our study models with a greater number of predictor variables provided better discriminatory capacity for the prediction of hospital mortality.1 When deciding whether a specific predictor variable should be included in a prognostic model, consideration must be given to the effort required to reliably obtain the data for that variable. In this regard, MPM III provides good discrimination using a small number of easily ascertained variables.

Drs Nathanson and Higgins write that “this study implies that DNR [do not resuscitate] status is not an important predictor of mortality in the ICU.” We believe this gives too much weight to the conclusions of our study. It has been previously documented that DNR status, by itself, is indeed a predictor of ICU and hospital mortality.2 Rather, DNR status did not—within the limits of our study as we discuss in the “Strengths and Limitations” section of our article1—significantly improve the performance of recent versions of the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS) models.

By “performance,” we refer to the commonly used and reported measures of prognostic model assessment, namely discrimination (measured by the area under the receiver operating characteristic curve) and calibration (measured by the Hosmer-Lemeshow statistic [HLS]). We agree with Drs Nathanson and Higgins’ belief that both measures are imperfect—although they are widely reported and used.3,4 As we discuss, calibration is especially subject to a variety of influences, including sample size and case mix. We further agree that a significant HLS does not necessarily mean that a predictive model is suspect, although a nonsignificant HLS is desirable. Calibration plots for each model showed discrepancies between observed and predicted values especially at the highest and lowest deciles of risk, consistent with poor calibration.

Recognizing the problems inherent in the assessment of model performance and the trade-off between discrimination and calibration, we also reported Brier scores for the models studied (Tables 2 and 3 of our article1). The Brier score, based on model prediction error, provides an overall estimate of performance. Our data demonstrated a progressive, albeit small, decrease in Brier score (ie, improved model performance) as model complexity increased. The addition of DNR status lowered the Brier scores for both APACHE models and for SAPS 3, suggesting its potential value as a predictor variable. As suggested, we also calculated both the Bayesian information criterion (BIC) and the corrected Akaike information criterion (AIC) for prognostic models with and without inclusion of DNR status.5 For each criterion, and similar to the Brier score analyses, the addition of DNR status was associated with a small improvement in model performance (approximately 3% decrease in AIC and BIC). The small improvements in Brier scores, BIC, and AIC associated with the addition of DNR status were not, however, reflected in statistically significant differences in area under the receiver operating characteristic curve or a consistent directional change in the value of the HLS.

References

Keegan MT, Gajic O, Afessa B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest. 2012;142(4):851-858. [CrossRef] [PubMed]
 
Azoulay E, Pochard F, Garrouste-Orgeas M, et al; Outcomerea Study Group. Decisions to forgo life-sustaining therapy in ICU patients independently predict hospital death. Intensive Care Med. 2003;29(11):1895-1901. [CrossRef] [PubMed]
 
Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604. [CrossRef] [PubMed]
 
Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. [CrossRef] [PubMed]
 
Akaike H. Likelihood and the Bayes procedure.. In: Bernardo JM, DeGroot MH, Lindley DV, Smith AFM, eds.Bayesian Statistics. Valencia: University Press; 1980:143-166.
 

Figures

Tables

References

Keegan MT, Gajic O, Afessa B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest. 2012;142(4):851-858. [CrossRef] [PubMed]
 
Azoulay E, Pochard F, Garrouste-Orgeas M, et al; Outcomerea Study Group. Decisions to forgo life-sustaining therapy in ICU patients independently predict hospital death. Intensive Care Med. 2003;29(11):1895-1901. [CrossRef] [PubMed]
 
Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604. [CrossRef] [PubMed]
 
Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. [CrossRef] [PubMed]
 
Akaike H. Likelihood and the Bayes procedure.. In: Bernardo JM, DeGroot MH, Lindley DV, Smith AFM, eds.Bayesian Statistics. Valencia: University Press; 1980:143-166.
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543