Over time changes in medical practice and therapy can result in improvements in hospital mortality. This can lead to systematic overestimation of mortality by prognostic systems based on 10-20 year old outcome data. Upon subsequent evaluation most ICUs’ performance will look good as the standardized mortality ratio (SMR) will be below 1.00, i.e. “grade inflation”. The purpose of this study is to compare results from two earlier versions of the APACHE III hospital mortality equation with a newly remodeled APACHE IV hospital mortality equation in order to examine how the performance of predictive models can change over time.
The APACHE III-h equation was based on outcomes from 16,662 patients admitted to ICUs during 1988-1989, and the APACHE III-i equation for 40,264 patients from 1993-1996. Each of these equations had good accuracy at the time of development. A new APACHE hospital mortality equation was built using patients admitted to an ICU during 2002 and 2003. The updated equation includes a new variable for patients who were sedated, adjusts for ventilation on ICU day 1, measures prior length of stay more accurately, and increases the number of disease groups from 94 to 116. The model was built on 66,270 patients and then validated on 44,288 patients.
The observed mortality was 13.51% and the predicted mortality was 13.55% yielding a standardized mortality ratio = 0.997 (p=0.79). The previous APACHE models when applied to the 2002-2003 data did not calibrate well. The predicted hospital mortality for version III-i was 14.64% for an SMR = 0.923 (p<0.001). Version III-h predicted hospital mortality to be 16.90% for an SMR = 0.799 (p<0.001). Thus, the older the model the worse its performance.
Prognostic models require repeated assessment and updating to adjust for changes in disease specific outcomes, to incorporate advances in statistical methods, and new knowledge about outcome prediction.
ICU performance comparisons using risk-adjusted hospital mortality should be based on models developed on contemporary data.
Andrew Kramer, Shareholder Cerner Corporation stock.; Employee Cerner Corporation.