0
Correspondence |

Resuscitation Status May Predict Mortality in Patients Admitted to the ICUResuscitation Status May Predict Mortality FREE TO VIEW

Brian H. Nathanson, PhD; Thomas L. Higgins, MD, MBA
Author and Funding Information

From OptiStatim, LLC (Dr Nathanson); and the Department of Medicine (Dr Higgins), Baystate Medical Center.

Correspondence to: Brian H. Nathanson, PhD, OptiStatim, LLC, PO Box 60844, Longmeadow, MA 01106; e-mail: brian.h.nathanson@att.net


Financial/nonfinancial disclosures: The authors have reported to CHEST the following conflicts of interest: Dr Nathanson’s company, OptiStatim, LLC, has an ongoing consulting agreement with the Cerner Corporation and participated in the development of the MPM-III model. Dr Higgins served previously as a consultant to Cerner and participated in the development of the MPM-III model. Dr Higgins also owns stock in Cerner.

Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details.


Chest. 2013;143(3):875. doi:10.1378/chest.12-2558
Text Size: A A A
Published online
To the Editor:

In the October 2012 issue of CHEST, Keegan et al1 compare three severity of illness scoring systems for critically ill patients (Acute Physiology and Chronic Health Evaluation [APACHE], Simplified Acute Physiology Score [SAPS], and Mortality Probability Model [MPM]) with and without a variable indicating the early presence of a do not resuscitate (DNR) order. They conclude that the model with the most variables (APACHE) had the best discriminatory ability, that adjusting for early DNR status did not improve model performance, and that all models were calibrated poorly. Although we agree with the authors’ first conclusion that more variables improve discrimination, we question the latter two.

This study implies that DNR status is not an important predictor of mortality in the ICU, even though it is significant in the MPM-III model. The authors saliently note that the SAPS 3 and APACHE IV models, which use more variables (20 and 142, respectively), may simply “capture” what predictive information DNR status conveys, making DNR status unnecessary. Unfortunately, the authors do not state if DNR status was statistically significant when added to the SAPS 3 or APACHE IV models.

Discrimination as measured by the C statistic (area under the receiver operating characteristic curve) indicates how well the model can separate survivors and nonsurvivors. If all nonsurvivors had higher predicted probabilities of dying than all survivors (regardless of the degree of predicted differences), then the C statistic would be 1, indicating perfect discrimination.2 Calibration as measured by the Hosmer-Lemeshow statistic measures how well predicted probabilities of mortality agree with observed numbers of deaths across deciles of risk.3

Cook2 has shown that because the C statistic is based on ranks, it is less sensitive than likelihood measures of model fit. Consequently, a strong predictor often causes minimal change to the C statistic in an existing model but can substantially change the predicted probabilities of those with the predictor. Instead, a Bayesian Information Criterion analysis or a related technique would be better suited to determine if a DNR status should be included in a model.

The Hosmer-Lemeshow statistic, like any inference test, is more likely to be statistically significant when the sample size is large. A significant Hosmer-Lemeshow test does not necessarily mean that a predictive model is suspect.4 Adjunct measures of model calibration, such as a calibration plot for each model, can be helpful when sample size is large.4

Finally, Cook2 has shown that there is a mathematical trade-off between good calibration and good discrimination, with well-calibrated models unable to achieve very high C statistics. Keegan et al’s1 study found that the MPM-III model had better calibration but worse discrimination than the APACHE or SAPS models. This does not suggest that one model is “better” than the others, but that each model has strengths and weaknesses and a more nuanced interpretation is required. Head-to-head comparisons of the three models are rare, and this article does fill a gap in the literature. However, more statistical analyses need to be done before one can definitively say that DNR status is not an important predictor of mortality or that these models are poorly calibrated.

References

Keegan MT, Gajic O, Afessa B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest. 2012;142(4):851-858. [CrossRef] [PubMed]
 
Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935. [CrossRef] [PubMed]
 
Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16(9):965-980. [CrossRef] [PubMed]
 
Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35(9):2052-2056. [CrossRef] [PubMed]
 

Figures

Tables

References

Keegan MT, Gajic O, Afessa B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest. 2012;142(4):851-858. [CrossRef] [PubMed]
 
Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-935. [CrossRef] [PubMed]
 
Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16(9):965-980. [CrossRef] [PubMed]
 
Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35(9):2052-2056. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543