Lessons for Continuing Medical Education From Simulation Research in Undergraduate and Graduate Medical Education: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines FREE TO VIEW

This article has four sections. The first defines medical simulation and summarizes presumptive findings about simulation-based continuing medical education (CME) from the Agency for Healthcare Research and Quality (AHRQ) Evidence Report.¹ The report aims to synthesize the results of nine literature reviews about “the effectiveness of simulation methods in medical education outside of CME.”¹ It also serves as a foundation for other sections of this article. The second section amplifies the findings from best evidence medical education (BEME) in one of the nine reviews.² The BEME review warrants special attention because it receives superficial coverage in the AHRQ Evidence Report,¹ addresses a different question about simulation in medical education beyond its effectiveness, is not confined to a single medical specialty or medical simulator, and offers practical advice about simulation- based medical education program planning and operation. The third section distills and presents lessons learned about best educational practices drawn from both the AHRQ report and the BEME review as well as from two graduate medical education (GME) and research programs and other published sources. The fourth section presents implications for CME grounded in the preceding narrative, the changing focus and professional character of CME, and observations about education for other learned professions.

Medical education using some form of simulation generally has been aimed at the junior trainee both for undergraduate medical education (UME) and for GME. The benefits of simulation derive from its standardization and reproducibility in contrast with the traditional apprenticeship approach to teaching where medical students and residents learn through practice with real patients in the clinic or hospital setting. With the increasing number of patients who are hospitalized and the shorter lengths of hospital stays, requirements for limited trainee work hours, and an emphasis on patient safety, simulation has received greater attention at the UME and GME levels. However, simulation seldom is discussed in the context of CME.

This article reviews the use of simulation education in baseline assessment of knowledge and skills, education grounded in learning objectives, intended outcomes expressed in metrics, deliberate practice with feedback, rigorous outcome evaluation, and professional accountability. These constructs are addressed thoroughly in a call for CME reform in the United States.³ This article urges the physician-learner to participate in CME activities that include deliberate practice and where he or she can work toward a mastery learning of CME objectives. Physician-teachers should design CME activities that make use of teaching techniques that assist the physician-learner in mastery learning and deliberate practice, embrace outcome measurement, and address cultural barriers to incorporate these educational approaches. Simulation as a teaching technique can be used by the physician-teacher to achieve these goals.

Medical simulation is defined as “a person, device, or set of conditions which attempts to present [education and] evaluation problems authentically. The student or trainee is required to respond to the problems as he or she would under natural circumstances. Frequently the trainee receives performance feedback as if he or she were in the real situation.”⁴ “Simulation procedures for evaluation and teaching have the following common characteristics:

Medical simulations are located on a continuum of fidelity, ranging from detached, multiple-choice examination questions;⁵ to more engaging task trainers (arms for phlebotomy practice); to full-body, computer- driven mannequins with sophisticated physiologic features that respond to pharmacologic and mechanical interventions.⁶ Simulations also include standardized patients who are live persons trained and calibrated to portray patients with a variety of presenting complaints and pathologies. Decades of experience and research demonstrate that standardized patients are highly effective for medical education and evaluation.⁷ Standardized examinees (students) also have been used as a way to calibrate and improve clinical skills examinations.^8,9 Medical educators have recently combined these modalities where standardized patients, inanimate models, and medical equipment are integrated to evaluate trainees' technical, communication, and other professional skills simultaneously.¹⁰

The AHRQ Evidence Report included a review of nine systematic reviews published between 1990 and 2006 that sought to evaluate the effectiveness of simulation methods in medical education outside of CME. The investigators abstracted data about study characteristics, educational objectives, learning outcomes, summary of results, conclusions, and quality of each review and graded the evidence of these articles according to each educational objective and outcome related to participant knowledge, attitudes, skills, practice behaviors, and clinical outcomes. The quality of each review was established using criteria derived from the Quality of Reporting of Meta-analyses statement,¹¹ which is intended for use only as a guide for preparing reports on quantitative metaanalyses that include randomized controlled trials.

The AHRQ report has several limitations, the most important is the review methodology, which failed to find two eligible reports.^12,13 This and other issues make the report's findings about simulation difficult to interpret unequivocally. Nevertheless, the AHRQ report¹ argues that the overall “direction of evidence points to the effectiveness of simulation training, especially for psychomotor skills (eg, procedures or physical examination techniques) and communication skills,” despite the low strength of the evidence “due to the small number of appropriate studies and scarcity of [reliable] quantitative data.” We add to these deficits the narrow focus of eight of the nine-included reviews (ie, single medical specialty, single simulation method) and the weakness of most primary studies covered in the reviews. These limitations are attributed, in part, to the lack of consensus about standardized methods to quantify clinical competence, a persistent problem in medical education research. The AHRQ authors also speculate that other limitations may include difficulty of establishing “clinical realism [high-fidelity] for participants,” and “other features that may be responsible for inadequate quality of evidence in this field.”¹

One of the nine literature reviews cited in the AHRQ report but not explained in depth is a systematic review done under the auspices of the BEME collaboration.¹⁴ The collaboration “involves an international group of individuals, universities, and organizations (eg, AMEE [Association for Medical Education in Europe], AAMC [Association of American Medical Colleges], ABIM [American Board of Internal Medicine]) committed to moving the medical profession from opinion-based education to evidence-based education. The goal is to provide medical teachers and administrators with the latest findings from scientifically grounded educational research.”²

The broad scope of the BEME systematic review² is distinct from the narrower focus on the effectiveness of simulation compared to other educational techniques. This article addressed best educational practices, reviewing 670 journal articles published between 1969 and 2003. Despite the original intent to conduct a quantitative metaanalysis, the studies were so heterogeneous and weak methodologically that the investigators resorted to a qualitative, narrative synthesis. The primary outcome of the BEME review is an inventory of 10 features and uses of high-fidelity simulations that lead to effective learning. These features are listed in order of reported frequency (percent) among the final BEME pool of 109 articles, and the report concluded that “the weight of the best available evidence suggests that high-fidelity medical simulations facilitate learning under the right conditions.² The 10 conditions² are as follows:

These findings, chiefly from UME and GME, are a guide for defining a research agenda on simulations as educational technologies. In contrast with the eight other literature reviews covered in the AHRQ Evidence Report, the BEME review spans a wide variety of medical specialties and simulation technologies across a long time frame. In addition, its emphasis on features and uses of medical simulation that lead to effective learning (eg, feedback, repetitive practice, curriculum integration), not just comparative effectiveness, sets a standard for understanding the benefits of simulation for medical education and personnel evaluation. The BEME review² on high-fidelity medical simulations also included a call for increased rigor in original simulation- based medical education research and improved journal-reporting conventions. In particular, the authors suggested that journal editors insist that all reports of primary research include basic descriptive statistics (eg, means, SDs, effect sizes, number of cases per group) that will permit quantitative synthesis in subsequent metaanalyses.

A subset of 31 journal articles reporting 32 research studies within the 109 articles was found to contain enough empirical data to permit a quantitative metaanalysis. The studies were framed to address the question, “Is there an association between hours of simulation-based practice and standardized learning outcomes?”¹⁵ Measured outcomes from these studies were cast on a standardized metric termed average weighted effect size. Hours of simulation-based practice in each study were grouped in the following five categories: none reported, 0 to 1.0, 1.1 to 3.0, 3.1 to 8.0, and 8.1 +. Data analysis revealed a highly significant “dose-response” relationship among practice and achievement, with more practice producing higher outcome gains. These results are presented in a subsequent report¹⁵ that demonstrated a direct relationship between hours of simulator practice and standardized learning outcomes.

The scholarship of the AHRQ report and BEME review is amplified by at least two other reviews^12,13 about simulation-based medical education that also can inform CME practices. One way to highlight advances in simulation-based UME and GME is to focus on exemplary education and research programs that identify their special features. Work completed by two medical simulation education and research programs^16–27 are illustrative, as they are thematic, sustained, and cumulative and are of special interest for chest physicians. Additional studies inform on academic standard setting²⁸ and mastery learning of clinical skills in advanced cardiac life support²⁹ and thoracentesis.³⁰

A more recent report³¹ discussed the “scope of simulation-based healthcare education,” pointing out that the best simulation-based medical education is a multiplicative product of simulation technology (eg, devices, standardized patients), teachers prepared to use the technology to maximum educational advantage, and curriculum integration. It argued that the major flaws in current simulation-based medical education stem from a lack of prepared teachers and curriculum isolation, not from technological problems or deficits.

The design of educational activities useful to practicing physicians assumes that CME program directors are knowledgeable about “what works” from scholarly reviews^2,12,13,15 and from individual studies having strong research designs, such as randomized trials,²⁴ mastery learning research,^29,30 and cohort studies.³² Program directors also should be informed about the latest scholarship on technology in medical education.³³ The key lesson is that medical simulation and other educational technologies are best used to complement, not replace, education grounded in patient care. CME in any form should be based on scientific best evidence rather than on opinion or habit.¹⁴

We endorse the position that CME best practices reside in educational programs that have the following three features: mastery learning, deliberate practice, and recognition and attention to cultural barriers within the medical profession that frustrate better CME programs. In particular, mastery learning and deliberate practice are ideally suited for simulation-based medical education. They also conform to accepted principles of adult education independent of teaching modalities.³⁴

Essential elements of the mastery learning model have been described in earlier publications.^35–37 In brief, mastery learning has the following seven complementary features:

The goal in mastery learning is to ensure that all learners accomplish all educational objectives with little or no variation in outcome. The amount of time needed to reach mastery standards for a unit's educational objectives varies among the learners. To illustrate, in mastery learning studies on acquiring advanced cardiac life support²⁹ and thoracentesis³⁰ skills, approximately 20% of the internal medicine resident trainees needed more time beyond the minimum allocation to reach mastery standards. The extra time needed was usually < 1 h.

The mastery learning model also includes other options in simulation-based education. For example, mastery learning can address learning objectives beyond skill acquisition to include knowledge gains; affective qualities, such as self-efficacy; or features of medical professionalism. Mastery learning requires a standardized curriculum for all learners, with uniform outcomes assessed by rigorous measurements and standards.^28,38,39

Deliberate practice is an educational variable associated with delivery of strong and consistent educational treatments as part of the mastery learning model.^40–43 Although demanding of learners, deliberate practice is grounded in information processing and behavioral theories of skill acquisition and maintenance.^40–42 It has at least nine requirements that can inform CME, as follows:

The goal of deliberate practice in a CME mastery-learning context is to require constant improvement of skill and knowledge rather than maintenance of a minimal level. Ericsson^40–42 cites data that underscore a “4/10 rule” about development of expertise in any field, as follows: it takes 4 h of deliberate practice every day for 10 years to become a world-class performer like an Olympic athlete, cutting-edge scientist, chess master, patient-care provider, or writer. Even Michael Jordan took 500 free throws every day throughout his professional basketball career to maintain and improve his professional edge.⁴⁴

Deliberate practice using medical simulation has been shown to improve medical performance in several medical and surgical specialties.⁴⁵ In a small study of pulmonary fellows trained with virtual bronchoscopy,⁴⁶ trainees demonstrated equal or better facility with an airway training model compared with skilled colleagues with several years of experience. This cascaded approach to education shapes “deliberate practice pedagogy.”⁴⁷ Joined with a mastery-learning program structure, the combination creates the potential for a more standard way to assess knowledge and competence.⁴⁸

It is unlikely that educational innovations involving simulation-based mastery learning and deliberate practice will receive a warm reception in the medical community. Innovations will be used effectively only after barriers in medical culture are acknowledged, addressed, and breached. Adoption of innovation in medical education at all levels has been slowed by inertia, habit, absence of audit and accountability, and other cited barriers. Medical education culture needs to embrace innovation and its diffusion as energizing opportunities that will boost professional competence, morale, and patient care. Specifically, barriers in contemporary medical culture that inhibit advancement of simulation-based CME are as follows:

Simulation technology as an educational tool could lead to significant changes in medical education, including a new emphasis on skill and knowledge acquisition and maintenance, integration of the technique into a comprehensive clinical curriculum that includes certification and recertification, adoption of mastery learning and deliberate practice, and increased competence and outcome measurement. Research should focus on valid and reliable tools for more systematic outcome measurements, with the ultimate goal of improving the quality of patient care. Policies that inform physician performance and govern the privilege to practice not only need to endorse the effective educational use of simulation technology, but also tackle sources of cultural resistance to its adoption.

Simulation will never replace the situational context and complex interactions learned through interaction with real patients. Expert mentors will always be needed not only to verify trainee performance in real situations, but also to judge the simulators' in vivo fidelity. Nevertheless, cultural barriers should not hinder the adoption and informed use of simulation technology as a powerful and effective educational tool to maximize physician and other health professional training and, ultimately, to improve patient care.

Dr. McGaghie has twice served as a paid speaker at Simulation Users Network (SUN) events sponsored by Laerdal Medical Corp.

Ms. Siddall has no conflicts of interest.

Dr. Mazmanian has no conflicts of interest.

Dr. Myers has no conflicts of interest.