0
Communications to the Editor |

Pitfalls in the Interpretation of Multivariable Models in the Critical Care Literature FREE TO VIEW

Aaron Joffe, DO
Author and Funding Information

Affiliations: St. Francis Medical Center, Honolulu, HI,  Veterans Affairs Palo Alto Health Care System, Palo Alto, CA Stanford University School of Medicine, Stanford, CA

Correspondence to: Aaron Joffe, DO, Staff Intensivist, St. Francis Medical Center, 2230 Liliha St, Honolulu, HI 96817; e-mail: doctrj@pol.net



Chest. 2005;127(1):411-412. doi:10.1378/chest.127.1.411
Text Size: A A A
Published online

To the Editor:

Recently, Barr and colleagues (April 2004)1 evaluated the effects of implementing a nutrition support protocol on the use of enteral nutrition, the time to the initiation of feedings, and the percentage of prescribed calories delivered. Duration of mechanical ventilation, ICU length of stay, hospital length of stay, and in-hospital mortality were studied as secondary end points. Using multivariable techniques, the authors concluded that their protocol increased the likelihood of receiving enteral nutrition and shortened the duration of mechanical ventilation, and that enteral nutrition reduced the risk of death. These conclusions depend on the integrity of the statistical analysis and, therefore, must be interpreted with caution.

Insofar as the ability of any model to predict or explain the studied outcome rests largely on the adjustment for variables that have been hypothesized on theoretic grounds or have been shown in previous research to be confounders of the relationship being studied,2the inclusion of age, gender, severity of illness, admitting diagnosis, and nutritional status alone hardly represents a comprehensive list of variables that have previously been shown to affect the primary and secondary outcomes of this study. The use of prokinetic drugs such as metoclopramide and erythromycin may facilitate feeding tube placement and improve gastric emptying, thereby improving tolerance to enteral feeding, decreasing the time to the initiation of enteral feedings, and increasing the percentage of prescribed calories delivered,3while ventilatory strategies limiting plateau pressures and employing lower tidal volumes have been shown to decrease mortality and to increase the number of ventilator-free days.4Furthermore, the reliability of the Cox proportional hazards model showing a 56% lower risk of death in those receiving enteral nutrition contains an insufficient sample size. As a general rule, there must be at least 10 outcomes for each independent variable that is eligible to be included.5 Thus, 70 patients would have to have died in order to accommodate the seven variables included.

Last, the study suffers from omissions in reporting on factors that that have been previously shown to make proper and comprehensive interpretation of published pulmonary and critical care literature difficult, if not impossible (ie, the name of the statistical package used, testing results for interaction or effect modification, the testing result for collinearity between independent variables, specifying the reason for choosing the variables included in the model, and goodness-of-fit testing).6

In summary, Barr and colleagues1 have shown that the implementation of a nutritional management protocol in the ICU is feasible. Unfortunately, deficiencies in the manner in which the authors express the multivariate analyses make any further conclusions regarding the effects of a nutrition support protocol difficult and underscore the need for acceptable guidelines or criteria for the reporting of multivariable analyses in order to allow readers to more accurately interpret the results of future studies.

References

Barr, J, Hecht, M, Flavin, K, et al (2004) Outcomes in critically ill patients before and after the implementation of an evidence-based nutritional management protocol.Chest125,1446-1457. [CrossRef] [PubMed]
 
Katz, M Multivariable analysis: a primer for readers of medical research.Ann Intern Med2003;138,644-650. [PubMed]
 
Booth, C, Heyland, C, Patterson, W Gastrointestinal promotility drugs in the critical care setting: a systematic review of the evidence.Crit Care Med2002;30,1429-1435. [CrossRef] [PubMed]
 
The Acute Respiratory Distress Syndrome Network. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome.N Engl J Med2000;342,1301-1308. [CrossRef] [PubMed]
 
Concato, J, Feinstein, A, Holford, T The risk of determining risk with multivariable models.Ann Intern Med1993;118,201-210. [PubMed]
 
Moss, M, Wellman, D, Cotsonis, G An appraisal of multivariable logistic models in the pulmonary and critical care literature.Chest2003;123,923-928. [CrossRef] [PubMed]
 
To the Editor:

We thank Dr. Joffe for his thoughtful comments regarding the use of multivariable models in critical care research that he wrote in response to our article that was recently published in CHEST (April 2004).1

Dr. Joffe expressed concern that we did not include several potential confounding variables in our multivariable models. While we agree that it is desirable to adjust for secular trends in management in a nonrandomized, interventional, before-and-after comparison such as ours, we limited the number of prespecified covariates in our models to avoid overfitting by using the rule of thumb that he describes in his letter (ie, at least 10 outcomes for each independent variable). Local practices for caring for critically ill patients, including ventilator management, were similar during the preimplementation and postimplementation phases of our study.

We disagree with Dr. Joffe’s comment about sample size. By definition, our study had sufficient power to demonstrate a statistically significant reduction in the risk of death in patients who were fed enterally.

At Dr. Joffe’s request, we are pleased to provide additional details about our methods and results. We selected covariates based on clinical judgment and entered all covariates into models without employing forward or backward selection procedures. We used Statview software (version 5.0.1; SAS Institute, Cary, NC) to perform all statistical analyses. Goodness-of-fit testing with the likelihood ratio and Pearson χ2 tests revealed that the model of enteral nutrition accounted for the outcome better than chance alone (p < 0.001), and that the predicted likelihood of the outcome was similar to the observed likelihood (p = 0.24). The results of the likelihood ratio and χ2 tests were similar for the model of mortality (p < 0.001 and p = 0.71, respectively). The correlation matrices of parameter estimates revealed no evidence of multicollinearity in models of enteral nutrition (R range, −0.14 to 0.23) or mortality (R range, −0.28 to 0.12). In post hoc analyses, we identified one statistically significant interaction between admission type and intervention group, such that the implementation of the nutritional support protocol had a greater effect on enteral feeding in surgical patients.

Based on this additional information, we hope that Dr. Joffe will agree that our conclusions are valid, namely, that the implementation of an evidence-based protocol for nutritional support resulted in a greater number of critically ill patients who were fed enterally and that enteral feeding was associated with improved survival.

References
Barr, J, Hecht, M, Flavin, K, et al Outcomes in critically ill patients before and after the implementation of an evidence-based nutritional management protocol.Chest2004;125,1446-1457. [CrossRef] [PubMed]
 

Figures

Tables

References

Barr, J, Hecht, M, Flavin, K, et al (2004) Outcomes in critically ill patients before and after the implementation of an evidence-based nutritional management protocol.Chest125,1446-1457. [CrossRef] [PubMed]
 
Katz, M Multivariable analysis: a primer for readers of medical research.Ann Intern Med2003;138,644-650. [PubMed]
 
Booth, C, Heyland, C, Patterson, W Gastrointestinal promotility drugs in the critical care setting: a systematic review of the evidence.Crit Care Med2002;30,1429-1435. [CrossRef] [PubMed]
 
The Acute Respiratory Distress Syndrome Network. Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome.N Engl J Med2000;342,1301-1308. [CrossRef] [PubMed]
 
Concato, J, Feinstein, A, Holford, T The risk of determining risk with multivariable models.Ann Intern Med1993;118,201-210. [PubMed]
 
Moss, M, Wellman, D, Cotsonis, G An appraisal of multivariable logistic models in the pulmonary and critical care literature.Chest2003;123,923-928. [CrossRef] [PubMed]
 
Barr, J, Hecht, M, Flavin, K, et al Outcomes in critically ill patients before and after the implementation of an evidence-based nutritional management protocol.Chest2004;125,1446-1457. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543