Simulation-Based Education Improves Quality of Care During Cardiac Arrest Team Responses at an Academic Teaching Hospital^*: A Case-Control Study FREE TO VIEW

The use of simulation technology has great potential to shape medical education, certification, licensure, and the quality of care.¹ Simulation has demonstrated its effectiveness to achieve, measure, and maintain trainee skill in the performance of a variety of clinical procedures including laparoscopic surgery,²³⁴ endoscopy,⁵ advanced cardiac life support (ACLS),⁶⁷⁸ emergency airway management,⁹ trauma resuscitation,¹⁰¹¹ bronchoscopy,¹² and carotid angiography.¹³ Although this technology has much promise, a key challenge is to link performance in the controlled simulation environment to the quality of the patient care delivered. Small studies have evaluated clinical performance after simulator training with encouraging results, but their scope is limited.^{9141516171819}

In July 2003, we initiated a simulation-based ACLS training program for internal medicine residents featuring 8 h of deliberate practice on a full-body patient simulator, immediate feedback, and rigorous outcome measures. We evaluated training outcomes using a randomized trial with a waiting list control group. Results showed that residents’ ACLS skills increased 38%⁶, were maintained over a period of 14 months,⁸ and that the training was rated highly as an adjunct to traditional didactic and clinical experience. The present study evaluates whether simulation-based ACLS education affects the quality of care during actual ACLS events at our institution.

This was a retrospective case-control study²⁰ of cardiac arrest team responses at Northwestern Memorial Hospital (NMH) from January to June 2004. An educational program had been started 6 months earlier for second-year internal medicine residents featuring simulation-based training in ACLS. A 6-month study period was selected because it provided an opportunity to compare two groups of ACLS team leaders as a natural quasi-experiment.²¹ Cases and controls are team responses to ACLS events divided into those who were directed by simulator-trained leaders vs those who were managed by traditionally trained leaders. The Northwestern University institutional review board approved the study and waived informed consent.

NMH is a tertiary health-care facility located in urban Chicago and is the principal setting for graduate medical education under the auspices of Northwestern University. At NMH, cardiac arrest teams, led by an on-call second-year or third-year internal medicine resident, respond to all “code blue” calls. All residents completed an American Heart Association (AHA) ACLS provider course at the beginning of the first year of training and again at the start of the third year of training (Fig 1 ).

Second-year internal medicine residents (ie, the simulator-trained group [n = 38]) received a 10-h simulation-based educational intervention in addition to routine clinical education and experience beginning in July 2003.⁶ As shown in Figure 1, the simulator-trained group underwent 1 h of baseline testing of ACLS skills, participated in four 2-h teaching sessions, and completed 1 h of posttesting at the beginning of the second residency year. All sessions took place in the NMH Patient Safety Simulator Center. The training program featured the life-sized human patient simulator (Medical Education Technologies, Inc; Sarasota, FL). This simulator operates using computer software and realistically portrays many of the physiologic and pharmacologic responses observed in ACLS situations. These features include responses of the respiratory system, pupils, and eyelids as well as heart sounds and peripheral pulses. Monitoring of noninvasive BP, arterial oxygen saturation, ECG, and arterial BP was utilized in the educational program.

The course faculty developed case studies and management checklists for the six most common ACLS scenarios encountered at NMH. These were asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia, and pulseless electrical activity. Educational content and assessment tools were based entirely on the ACLS Provider Manual²² used by the AHA as instructional materials for ACLS provider courses. Teaching sessions allowed groups of two to four residents time to practice the protocols and procedures, and to receive standardized education from simulator faculty. Debriefing allowed the residents to ask questions, review algorithms, and receive feedback. The four teaching sessions were presented in uniform order as follows: (1) procedures (intubation, central line placement, pericardiocentesis, and needle decompression of tension pneumothorax); (2) pulseless arrhythmias (asystole, ventricular fibrillation, and pulseless electrical activity); (3) tachycardias (supraventricular and ventricular); and (4) bradycardias (second-degree and third-degree atrioventricular block).

Third-year residents (ie, the traditionally trained group [n = 40]) had more postgraduate training and had recently renewed their ACLS provider status through an AHA ACLS course, as shown in Figure 1, but were not trained using the simulator. The study period provided a unique opportunity for a natural quasi-experiment²¹ to compare these two groups.

A log of cardiac arrest team responses is maintained by the NMH Cardiopulmonary Resuscitation Quality Committee. This log was used to identify the 72 cardiac arrest team responses during the 6-month study period. All 72 medical records were available for review. Twenty-two medical records were excluded because they were respiratory arrests primarily managed by anesthesiology or were non-ACLS situations such as near syncope, chest pain, or asymptomatic hypotension. Two records were excluded because the cardiac arrest flowsheet could not be obtained.

A sequential review of the remaining 48 medical records was conducted. Data were abstracted from the cardiac arrest flowsheet completed at each ACLS event by an ICU nurse trained and assigned by the Cardiopulmonary Resuscitation Quality Committee. Because the primary study reviewers were not blinded to residents’ training status, 20 randomly selected records were re-reviewed by a senior hospitalist faculty member who was blinded to residents’ training status and the results of the first review. Performance ratings from the second reviewer were compared with ratings by the primary reviewers to assess interrater reliability.

Mirroring the simulation training evaluations,⁶ medical abstraction was based on the 2003–2004 AHA guidelines, which list four key steps across six common ACLS events (ie, asystole, ventricular fibrillation, supraventricular tachycardia, ventricular tachycardia, symptomatic bradycardia, and pulseless electrical activity) to compute the total correct performance score as a percentage. The four measures were selected because they were common across ACLS scenarios, did not depend on physician judgment, could be scored objectively, and were based on algorithms from the AHA ACLS textbook.²² The four key steps are as follows:

For the most common condition, refractory ventricular fibrillation (n = 16), a more detailed assessment was possible due to the AHA algorithm for this scenario.²² The algorithm consisted of the following: (1) use of defibrillation immediately after administering basic life support; (2) administration of three shocks without interruption; (3) use of a drug-shock-drug sequence; and (4) administration of an antiarrhythmic drug at the appropriate point in the algorithm. Abstracted data also included patient age and pre-cardiac arrest ventilator and telemetry status.

Adherent responses to each ACLS event were defined as compliance with AHA guidelines of ≥ 75%, based on adding all possible correctly performed performance measures. This cutoff was used because it is identical to the mean of the passing performance standards set by an expert panel in an earlier ACLS standard-setting study.²³

Interrater reliability was assessed using the κ coefficient²⁴ adjusted using the formula of Brennan and Prediger.²⁵ Group differences between simulator-trained and traditionally trained residents were evaluated using the t test and the χ² test. Logistic regression was computed to determine the odds ratio of an adherent ACLS response, testing the difference between groups controlling for patient age, and preevent ventilator and telemetry status (each coded “Y”/“N”). Logistic regression variables were selected because they were documented in the medical record and had been used in prior studies²⁶²⁷ of survival from in-hospital cardiac arrest events.

The mean interrater reliability for the 20 randomly selected ACLS events across the eight performance measures was high (κ = 0.87). This indicates that there was very little disagreement about abstraction of ACLS performance data.

Twenty ACLS events were led by simulator-trained second-year residents and 28 were led by traditionally trained third-year residents. As shown in Table 1 , no difference between patients in either group was shown by age, ventilator, or telemetry status.

Simulator-trained residents showed a significantly higher adherence to AHA standards (mean correct responses, 68%; SD, 20%) vs traditionally trained residents (mean correct responses, 44%; SD, 20%; t_[46] = −3.7; p ≤0.001) [Fig 2 ].

Using the ≥75% cut point for an adherent performance, 13 of 20 ACLS responses (65%) led by simulator-trained residents achieved this standard vs 6 of 28 (21.4%) responses led by traditionally trained residents (χ²_[1] = 7.7; p ≤0.01). Simulator-trained residents were over seven times more likely to lead an adherent ACLS response than traditionally trained residents (odds ratio, 7.1; 95% confidence interval, 1.8 to 28.6) after controlling for patient age, ventilator, and telemetry status in logistic regression.

Twenty-two of the 48 patients survived the ACLS event, and 3 patients survived to hospital discharge. Nine of the 20 patients (45%) managed by simulator-trained residents survived the event and 13 of the 28 patients (46.4%) managed by traditionally trained residents survived the event (p = 0.92). Although postevent survival was not different between groups, the mean survival time to death or hospital discharge among the nine surviving patients managed by simulator-trained residents was 194.7 h compared to 107.1 h for the 13 surviving patients managed by traditionally trained residents (p = 0.11). Two of the patients surviving to hospital discharge were in the simulator-trained group (discharge rate, 10%), and one patient was in the traditionally trained group (eventual discharge rate, 3.6%; p = 0.36).

This study amplifies the outcomes of past research on simulation-based ACLS medical education by extending those research findings to the quality of care provided during real ACLS events. Our prior work documented the ability of simulation-based training to significantly increase residents’ ACLS skills,⁶ to allow for the setting of mastery standards,⁷ and to demonstrate the retention of skill over a 14-month study period.⁸ The research situation described in this article presented a unique opportunity to assess clinical responses of simulator-trained vs traditionally trained residents in an actual patient care environment. Our study effectively demonstrates the ability of a simulation-based education program to improve adherence to AHA ACLS guidelines in a group of residents, even though they had less clinical experience and less traditional ACLS training (one vs two provider courses) than the comparison group (Fig 1).

While supporting a role for simulation training grounded in deliberate practice, our results highlight an unanswered question, “Can competence be evaluated independent of outcomes?”²⁸ The clinical process was fulfilled by the ACLS training and evaluation program, but differences in clinical outcome were not achieved because most patients died regardless of the treatment they received. Our survival to hospital discharge rate of 3 of 48 patients is within the range of other published studies²⁹ and reflects the high prearrest morbidity of this patient population. Many factors may also contribute to survival from in-hospital cardiopulmonary arrest including patient characteristics, initial cardiac rhythm, and circadian variation.²⁹³⁰ Given these cofounders and our small sample size, it is not surprising that our results did not conclusively show improved survival as a result of simulator training. Whether this would be seen in a larger patient sample is unknown, although a trend toward longer postevent survival and an improved survival to hospital discharge rate were seen.

Many of the ACLS events led by our residents did not meet the minimum standard for resuscitation despite prior conventional training and experience. This finding is consistent with those of other studies that show clinical experience alone is often insufficient to ensure the acquisition of such basic clinical skills as chest radiograph interpretation,³¹ hypertension diagnosis and management,³² cardiac auscultation,³³ and emergency airway management.³⁴ Poor adherence to AHA guidelines by well-trained hospital staff during in-hospital ACLS events³⁵³⁶ and a rapid decline of skill after traditional ACLS education³⁷ have also been well documented. Our results show a dramatic improvement in adherence to accepted guidelines in the clinical setting among residents who had an opportunity for focused, deliberate practice in a controlled simulation environment. Our prior observations⁸ of long-term retention of these skills following simulator training are encouraging and warrant further study.

Our study has several limitations. It represents a small sample of events at a single institution over a short time period. It used a case-control retrospective design rather than a prospective randomized design. The possibility of omissions or errors in chart documentation cannot be excluded. Clinical factors other than the training status of team leaders, such as input or cuing from other members of the resuscitation team, may influence adherence to ACLS guidelines. The existence of other confounding or uncontrolled factors cannot be definitively ruled out. However, the magnitude and consistency of the statistical results lead us to believe that the skill improvements gained in the simulated environment⁶⁷⁸ can be generalized to the clinical setting and are not an artifact of chance or other variables.

In conclusion, the use of a simulation-based training program improved the quality of resuscitation efforts by internal medicine residents in actual ACLS events at our institution. We believe traditional bedside and clinical teaching in medical education should be amplified to include simulation-based training. This may be especially useful in high-acuity skills such as ACLS, which are required for board certification³⁸ yet occur infrequently.²⁷ In patients with critical illness, teams of physicians need to respond effectively to provide high-quality patient care. This study adds to the growing body of knowledge that simulation-based education improves the procedural skills of physicians^{234567891011121314151617181934} and may enhance the quality of their patient care.^{914151617181934} In this study, deliberate practice is shown to be a powerful tool to boost the competence of physicians and the quality of their patient care in actual ACLS events.

We thank the Northwestern University internal medicine residents for their dedication to patient care and education. We acknowledge Charles Watts, MD, and J. Larry Jameson, MD, PhD, for their support and encouragement of this work. We appreciate the input of anonymous reviewers who helped to strengthen this article.