0
Correspondence |

Serious FunSimulation Testing and the CHEST Challenge: Adding Summative Simulation-Based Testing to the CHEST Challenge FREE TO VIEW

William F. Kelly, MD, FCCP; Ed Dellert, RN, MBA; Mark Rosen, MD, FCCP
Author and Funding Information

From the Department of Medicine (Dr Kelly), Uniformed Services University of the Health Sciences; American College of Chest Physicians (Mr Dellert); and Pulmonary, Critical Care, and Sleep Medicine (Dr Rosen), North Shore University Hospital.

Correspondence to: William F. Kelly, MD, FCCP, Department of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Rd, Bethesda, MD 20814; e-mail: Wkelly@usuhs.mil


Financial/nonfinancial disclosures: The authors have reported to CHEST the following conflicts of interest: Dr Kelly serves as Chair of the CHEST Challenge. Mr Dellert is a full-time employee of the American College of Chest Physicians. Under his direction, CHEST Challenge 2010 was integrated with the educational opportunities provided during CHEST 2010, including the simulation center. These areas are developed and implemented using a team-based approach from the committee leaders and instructional design staff of the American College of Chest Physicians. Dr Rosen served as Master of Ceremonies at the CHEST Challenge competition from 2008 to 2010.

Reproduction of this article is prohibited without written permission from the American College of Chest Physicians (http://www.chestpubs.org/site/misc/reprints.xhtml).


© 2011 American College of Chest Physicians


Chest. 2011;140(1):267-268. doi:10.1378/chest.11-0174
Text Size: A A A
Published online

To the Editor:

In a recent issue of CHEST (January 2011), Khouli et al1 confirmed the power of simulation-based education for assessing trainee performance in a risk-free environment and improving that performance compared with traditional apprenticeship or video training. There was even an association with a clinical outcome, namely, reduced catheter-related blood stream infections, although the patient outcome control groups were historical or in different populations (surgical ICU vs medical ICU).

For educators starting or improving their own simulation-based medical training programs, the right conditions and best practices from the Agency for Healthcare Research and Quality and Best Evidence in Medical Education Collaborative are reviewed in a CHEST supplement.2 Use of simulation for summative, high-stakes evaluation, such as certification and licensing examinations, is increasing.3 Some suggestions for designing one’s own summative evaluations are introduced here.

CHEST Challenge4 is the annual international contest of the American College of Chest Physicians in which teams of fellows in training compete by answering questions online and in live game show-style experiences. In 2010, skills assessment using high-fidelity medical simulation was added, constituting 10% to 20% of the total points awarded.

Evidence for defining skills, developing metrics, and evaluating reliability and validity of these assessments has been described.3 Prior to beginning the simulation component of the CHEST Challenge, the facilitator and graders practiced the clinical scenarios and agreed on standardized scoring by consensus. Players were oriented to the capabilities and limitations of the mannequin. Graders utilized valid, behaviorally anchored checklists (core actions were either present or not), although holistic (global) scoring also has value. Two graders were used for interrater reliability, but adding simulation tasks (broader domain sampling) may best improve overall reliability. Generalizability studies, if done, can further assess the sources and magnitude of measurement errors and help with test design. A trained facilitator played the role of the ICU nurse to ensure consistency and respond to participants for all technical limitations (eg, simulator does not sweat or have changes in skin color). As much as possible, player participants had to perform rather than just verbalize interventions. Finally, encounters were recorded on video; although intended for promotional value and quality assurance and not necessarily considered better than oral debriefing,5 these recordings can provide learners with valuable feedback and insights.

In a post-simulation anonymous survey (24/27, 89%), 58% of players indicated that they had used simulation equipment at their home institutions (usually for bronchoscopy or advanced cardiac life support training); 42% had participated in a simulation activity of the American College of Chest Physicians, and 29% did so elsewhere. Using a 5-point Likert scale, fellows responded that they were very comfortable with the simulator (mean ± SD, 3.96 ± 0.85; 71% positive vs 4% negative responses). Most enjoyed the testing (3.90 ± 1.06, 67% vs 13%) and believed that it was fair (3.96 ± 1.20, 75% vs 17%), and the majority would like even more added to next year’s CHEST Challenge (3.63 ± 1.35, 54% vs 21%). Some recommended additional exposure to the simulator prior to testing. Identifiable player information would be required to assess criterion-related validity.

Since 2002, CHEST Challenge has offered a fun forum, rewarding fellows for their medical knowledge and professional attitude. With more experience and guided by best evidence, we believe that summative simulation-based testing also will allow us to measure skills both in our game and in our fellowship training programs.

Khouli H, Jahnes K, Shapiro J, et al. Performance of medical residents in sterile techniques during central vein catheterization: randomized trial of efficacy of simulation-based training. Chest. 2011;1391:80-87. [CrossRef] [PubMed]
 
McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. American College of Chest Physicians Health and Science Policy Committee American College of Chest Physicians Health and Science Policy Committee Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135suppl 3:62S-68S. [CrossRef] [PubMed]
 
Boulet JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med. 2008;1511:1017-1024. [CrossRef] [PubMed]
 
American College of Chest PhysiciansAmerican College of Chest Physicians CHEST Challenge Web site.http://www.chestchallenge.org. Accessed January 19, 2011.
 
Savoldelli GL, Naik VN, Park J, Joo HS, Chow R, Hamstra SJ. Value of debriefing during simulated crisis management: oral versus video-assisted oral feedback. Anesthesiology. 2006;1052:279-285. [CrossRef] [PubMed]
 

Figures

Tables

References

Khouli H, Jahnes K, Shapiro J, et al. Performance of medical residents in sterile techniques during central vein catheterization: randomized trial of efficacy of simulation-based training. Chest. 2011;1391:80-87. [CrossRef] [PubMed]
 
McGaghie WC, Siddall VJ, Mazmanian PE, Myers J. American College of Chest Physicians Health and Science Policy Committee American College of Chest Physicians Health and Science Policy Committee Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135suppl 3:62S-68S. [CrossRef] [PubMed]
 
Boulet JR. Summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med. 2008;1511:1017-1024. [CrossRef] [PubMed]
 
American College of Chest PhysiciansAmerican College of Chest Physicians CHEST Challenge Web site.http://www.chestchallenge.org. Accessed January 19, 2011.
 
Savoldelli GL, Naik VN, Park J, Joo HS, Chow R, Hamstra SJ. Value of debriefing during simulated crisis management: oral versus video-assisted oral feedback. Anesthesiology. 2006;1052:279-285. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543