The Science of Continuing Medical Education: Terms, Tools, and Gaps: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines FREE TO VIEW

In preparing this systematic review and its subsequent report, the evidence-based guideline (EBG) committee discovered several elements in the continuing medical education (CME) research literature that, if addressed, would advance our understanding of the mechanisms by which CME works and the best means of studying them. The committee's effort was aided by a review of the methods and findings of the evidence-based practice center (EPC) report, in addition to input from the two principal investigators of the report (S.S.M. and T.D.). They used their knowledge of the details of the methods to help the committee prepare a rigorous critique of the report and to provide guidance for future researchers. The process of reflection, self-study, and improvement is the hallmark of any discipline¹; it can inform the practice of CME and by doing so, can effect changes in practice performance and possibly health care outcomes.²

This article is divided into two sections. The first section addresses the process and findings of the EPC report, noting its strengths and weaknesses, and resultant limitations to the conclusions of the report. The second section attempts to look forward and to construct a more robust approach on which to build future research in CME.

Implications to the physician-learner and physician- teacher are significant. From a physician-learner perspective, CME has been and still is considered as a means to participate in short courses or conferences in order to fulfill a time-based credit system. In recent years, the American Medical Association (AMA) and other certification boards have broadened the types of CME activities that the physician can and should participate in. From a physician-teacher perspective, this will only lead to increases in the need for and quality of CME activities in the future, which will place more rigor and scientific demand on physician-teachers. Future research in CME is needed, especially that requiring a rigorous process of retrieving, extracting, and synthesizing the data from CME activities and the literature.

As noted elsewhere in this supplement, the heterogeneity of the studies and the lack of a clear understanding and description of the interventions limited interpretation of the data and subsequent recommendations. To increase the rigor of such studies and to permit more appropriate comparisons among them, we considered the methods and findings of the EPC in the following three areas: its use of nomenclature, its systematic review process, and the nature of the primary literature itself.

There are several reasons for confusion in the application of the “language” of CME. First, the widely held perception of the acronym CME generally is one of the short course or conference and the credit systems that attend it. However, many definitions, such as that provided in the next paragraph by the AMA, encompass a much broader scope. Second, studies in physician learning and change and the context in which they occur have been enriched, although possibly made more complex, by the addition of scholarship from diverse disciplines. For example, educators might use the term outreach visits or distance education to describe education in remote site visits, whereas health services researchers might call the same intervention academic detailing. Third, new, parallel fields of study (eg, in health services research) have used terms like implementation tools to describe a wide variety of strategies (eg, reminders at the point of care, audit and feedback, among others) that some may consider CME techniques. These discrepancies lead us to call for a standard set of definitions in developing and comparing interventions and methods. These definitions attempt to describe interventions and techniques in CME; their intent is to be comprehensive, not exhaustive, and they are not meant to describe the learning process undertaken by physicians and others.

The AMA³ defines CME as all “educational activities that serve to maintain, develop, or increase the knowledge, skills, and professional performance and relationships a physician uses to provide services for patients, the public, or the profession.” Further, the AMA³ defines the content of CME as “that body of knowledge and skills generally recognized and accepted by the profession as within the basic medical sciences, the discipline of clinical medicine, and the provision of health care to the public.”

However, within this broad framework exists a wide variety of strategies, interventions, and techniques that require clarity of definition for the readers of these guidelines, including CME providers, physicians, policymakers, government regulators, funders, and others. In an attempt to develop a common language for those involved in the research and practice of CME and to set the stage for future research and understanding in this field, we provide here an array of definitions. To accomplish this goal, the EBG committee first reviewed the definitions used by the investigators at Johns Hopkins in preparing the EPC report. Where the committee agreed that these definitions were appropriate, they were maintained; where a definition appeared insufficient or confusing, we developed or used alternate definitions, borrowing heavily from existing texts and other resources.^4–9 A listing of final proposed definitions is provided in Appendix 1 of the EPC report.

The EPC review used two terms that overlapped. The review defined media methods as the means by which information is conveyed (eg, live, Internet, print) and educational techniques or methods as strategies to be used within the context of media (eg, case discussion in a live presentation). The EBG committee searched for alternative classifications and definitions that might clarify these educational constructs. Three sources were useful in distinguishing instructional models, strategies, methods, and media when looking at instructional practices.^4–6

Based on these three sources, we recommend the following framework and definitions:

The design of effective instruction is based on the educational needs and goals of the learners that guide the particular mix of instructional models, strategies, methods, and media used. Accordingly, the EBG committee followed the definition of the EPC and construed the term multimethod as any combination of individual forms of instructional methods (eg, case-based simulation with discussion and opinion leaders) and multimedia as the combination of individual forms of instructional media (eg, text, speech, images).

In addition to broad classifications, the EBG committee noted several definitions that deserve clarification. First, some terms in the report were constructed too narrowly (eg, audio referred to the use of audiotapes alone); too broad (eg, handheld referred to both laminated cards and personal data assistants, which are quite different media in terms of capacity and use); or tautological (ie, using the same word [eg, attitude] to define itself). Second, simulations were considered as separate entities in the EPC review; the EBG committee considers this instructional method integral to the CME process and is presented as such in the definitions. Finally, the committee considered a broader understanding of the term outcomes than those described in the EPC report (methodologic issues are detailed later). For example, Kern et al⁷ distinguish the following three types of objectives that could be seen here as end points in the CME process: learner, process, and outcome. In this sense, process may refer to practice or performance change and outcome to clinical, patient end points or measures, the final construct of which also may include families and population. Other taxonomies of outcome exist as well. Miller, Dixon, and Kirkpatrick (in the QUOROM statement)¹² each refer to knowledge, competence, and performance as outcomes, whereas Bloom's taxonomy of cognitive, affective, and psychomotor measures of learning also may prove useful in determining the outcomes of educational interventions.

Specific definitions are listed in Appendix 1 of the EPC report, and an attempt to clarify the description of outcomes is made in the Continuing Healthcare Education Study Template (CHEST), which is described later. We used existing definitions whenever possible, acknowledging that there is some overlap among them.

Research in CME and related areas will require rigorous primary studies (the subject of the last section in this article) and an equally rigorous process of retrieving, extracting, and synthesizing data from this literature. The guideline briefly summarizes the major steps undertaken in this review, touching on some examples of best practice and on possible weaknesses and limitations, and suggests ways in which the literature and its retrieval and synthesis could be improved. The Methodology article describes the methods the EPC employs in detail. In short, the review undertook the following three major steps: identifying the primary and systematic review literature, selecting studies for review, and abstracting data and assessing study quality.

In preparing the EPC report, a sizable amount of literature was retrieved and reviewed with a relatively high degree of consistency, standardization, and rigor. The EPC report and process expanded on previous reviews of CME effectiveness and allowed for the formulation of an EBG, incorporating recommendations by the American College of Chest Physicians guideline committee. The EBG committee noted several particular strengths of the report and its process in addition to the standard process of most systematic reviews in this area. The committee remarked on the efforts of the EPC to rate study quality, basing it on the criteria of Jadad et al.⁸ In addition, the attempt to characterize and capture features of adult learning as features of CME studies are unique and noteworthy, as were its efforts in the area of attempting to document the “dosing” of the CME intervention—its duration, frequency, and intensity. Further, the EPC's attempt to model the way in which CME works is useful and beyond the work of most systematic reviews and contributes sizably to the field. Finally, the EBG committee endorsed the inclusion of a panel of CME experts who advised on the overall process of the EPC's review.

Despite its many strengths, the EPC report contains several limitations that preclude generalizing its findings. In articulating these limitations, we note the EPC's assessment that the wide scope, limited resources, and strict timelines in this review necessitated “hard methodologic choices” regarding the scope of the key questions and extent and nature of the initial search strategy.

First, the EBG committee noted that the CME literature, especially as it applies to quality improvement, cost-effectiveness, and other dimensions of its place in healthcare, occupies terrain not always identified by the search strategies outlined in this methodology, and the EPC could have employed more comprehensive databases already created for this purpose. First, the University of Toronto's Research and Development Resources Base in CME⁹ and the Best Evidence Medical Education Collaboration¹⁰ could provide substantial contributions in future systematic reviews. Second, restricting the search to the United States and Canada excludes a sizable body of research generated in other countries with very similar CME and training requirements, including the United Kingdom, Australia, New Zealand, and the Netherlands. Third, the formal US CME provider accreditation process regulates the provider but does not address the manner, methods, or instructional techniques of education, the subject of this report. Fourth, nonrandomized trials with comparator groups were included, allowing volunteer bias that may skew results. Fifth, limiting the search to those studies involving 15 or more physicians excludes studies with fewer subjects but adequate statistical power, as in small group or individualized learning; these are important, but often-neglected areas of CME. Finally, the EPC did not include specific search terms such as clinical practice guideline, performance practice, and quality improvement.

As discussed, the process of data retrieval and documentation may have been affected by the lack of definitional agreement in the primary literature, making the data abstraction process difficult. For example, the EPC review assessed data about duration and frequency of exposure to CME activity, but lack of standardization in the reporting of such data precluded meaningful interpretation. Further, given that the review process was sequential, no efforts could be made to measure and achieve high degrees of interrater reliability between data abstracters and reviewers. The sequential review may have introduced a subjective element into the review of the data, with the potential for variability in data abstraction and interpretation compared to an independent, double-review process.

The committee was concerned about first the extent to which the EPC was able to describe and document study outcomes. The differentiation between primary and secondary study objectives and educational or learning objectives lacked clarity. Although most differences may have been minimal, these may have rendered data abstraction less precise and permitted the possibility that achievement of study objectives of low importance (eg, a secondary study objective) could trump the failure to achieve other objectives of greater clinical or educational significance. Second, the EPC's determination of the degree to which studies met (or failed to meet) their objectives lacked clarity. Studies were determined to have met their objectives by a simple answer of “yes” when most study objectives were met and “no” when not met (see Methods^7a). The word mixed was used to describe an intermediate state, and the word unclear was used when the study design did not permit interpretation. In some instances, the primary literature made interpretation difficult. Third, in some cases of self-reports (19 of 135, or 14%), performance outcomes susceptible to subjective interpretation by the respondent,¹¹ were counted as measures of performance. In this context, we suggest that to assure unbiased assessment, reviews of CME assessing physician performance use only objective measures. Finally, direct comparisons among many studies that used similar interventions and statistical measures were possible. Here, the determination of effect size, where feasible, would have contributed to the body of knowledge about CME effectiveness. Taken together, these strengths and limitations of the review process are important to consider in the interpretation of the EPC report and lead to recommendations regarding further research.

This EPC review included considerations of study quality, clearer articulations of which may inform future research in this area. Quality assessment of trials was based on the criteria of Jadad et al,⁸ noting that because participants in a study of an educational intervention cannot be blinded to the intervention, trials were assessed for evidence that the outcomes evaluation was blinded. Further, the EPC review reported on the validity and reliability of the methods used to measure the effects of CME relative to its selected outcomes measures. The quality of each systematic review was assessed using a tool derived from the main elements of the QUOROM statement¹² as a basis, with the addition of questions regarding assessment of publication bias.

Analysis of the effect of a discipline requires a clear and consistent methodology applied at the primary research level and a rigorous process of searching, retrieval, distillation, and synthesis of this primary literature. A clear statement of the expectations of scholars and other experts about the nature and description of primary research in this area is essential, much as in the development and reporting of clinical practice guidelines.¹³ We applaud the EPC's attempt to move the science of CME forward by quantifying exposure to educational activities regarding duration and frequency and by determining and using strict inclusion criteria for systematic review. The EBG committee acknowledges the limitations of synthesizing the CME literature, given the heterogeneous nature of the primary studies and their differing audiences, methods, and content areas. Further, we concur with the EPC's assessment of the low quality of study designs, at least as they were defined by this process. Factors that lead to these low-quality studies include variable reporting of details, dearth of models or conceptual frameworks, critical reviews of past studies on which to base research, and the lack of valid and reliable evaluation tools. Moreover, the EPC review reveals a relatively nonstandardized approach to CME research in general.

Finally, the EPC attempted to analyze the studies according to their attention to some of the principles of adult learning, elsewhere called adult education. These principles included enabling learners to be active contributors to their learning, relating to learners' current work or life experiences, tailoring curricula to learners' current or past experiences, allowing learners to identify their own learning goals and direct their education, allowing them to practice what they learned in simulated activities, providing support to self-directed learners, receiving feedback from teachers or peers during active learning, allowing learners to reflect on their learning and to observe faculty role modeling. In general, however, this analysis was lacking or was not reported.

Despite its limitations, recommendations in this guideline were made based on the EPC review with some confidence. In contrast, there were many other areas in which the small number and heterogeneity of available studies precluded reaching definitive conclusions regarding the influence of particular factors on the effectiveness of CME. In this section, a broad framework is provided on which further research can occur, describing a theoretical model of research based on the EPC report. We recognize that such a model is an option but believe that this framework may suffice to characterize research elements and to describe their interrelationship. Further, we attempt to describe a template for designing and reporting trials of CME interventions as a basis for discussion.

One framework guided the work of the EPC itself. The EBG committee, using this model and others,^19–21 attempted to develop a framework for further research. Such a framework, frequently referred to as knowledge translation, would encourage investigators to address broad areas or variables (eg, the learner) as well as issues within each variable (eg, age, career stage, specialty, gender). Further, such a platform would enable the testing of the ways in which the variables interact and advance the CME research agenda.

In addition to considering the learner, the educational intervention, the practice, and external environment, we consider the Rogers²² construct of innovation (eg, in this case, new knowledge or skills) to be important. The following characteristics must be considered as innovations are adopted: the nature of the evidence behind the information or change; its complexity; whether it can be observed and tested; the advantage of obtaining this information; and the size of the change (ie, small changes, such as a minor adjustment to a method or clinical strategy; large changes requiring a major shift in knowledge or competence, such as acquiring a brand-new manual skill).

We propose that a similar framework be employed whenever complex CME studies of effect are undertaken. To enable this process, the committee described the components of each category, outlined in checklist format, in Table 1 (CHEST). In order to frame the research agenda, the guideline committee suggested a number of questions that pertain to each variable. Finally, there are many overarching questions, among them the interplay among the following variables.

Do the attributes and credibility of the educational provider have an effect on learning and change? What are the variable effects of the CME medium, method, and technique used? What are the effects of the setting of the CME activity on learning? What are the effects of the duration, intensity, and frequency of the intervention or by its attention to adherence to theories and principles of learning?

What are the effects of age, training, specialty, practice setting, and manner of reimbursement on learning and change?

What are the effects of external regulations, incentives, disincentives, public or patient demand, practice setting, and the presence of teams on learning, performance, and health outcomes? What are the effects of physician payment systems, including pay for performance? What are the effects of access to information technology resources at the point of care?

What are the effects of such elements as cost, ability to be observed and tested, complexity, level of the innovation or information conveyed by the CME activity?

The EBG committee's review of the EPC report leads to a call for further clarity about the nature and elements of CME research. In this effort, the committee developed the CHEST. The template attempts to help ensure that researchers attend to and report the details, among others, of the appropriateness and use of study methods; the participants, including their settings, workload, and other practice considerations; intervention media, methods, and techniques; outcome measures, units of analysis, and allocation; follow-up of participants; the quality, blindedness, and reliability of evaluation measures; protection against contamination; ethical approval; and targeted clinical behavior and complexity of change. We expect that further deliberation (see recommendation 11–8) will lead to statements of optimal study design, such as those created for other purposes (eg, QUOROM¹²); the development of guidelines in this area (eg, the Appraisal of Guidelines Research and Education guideline appraisal instrument¹⁴); and reporting of the strength of findings (eg, Scottish Intercollegiate Guidelines Network,¹⁵ Grading of Recommendations Assessment, Development, and Evaluation¹⁶). For the present, the committee proposes the CHEST checklist as a beginning to add rigor to study design and reporting.

Designing a proper study requires considering the complexity and interplay of variables measured quantitatively, the exclusion of nonquantitative methods, and the use of randomized control trial methodologies, while at the same time understanding the limitations of these methods. For example, the EBG committee noted the natural focus on measurable end points, such as prescribing or test ordering, and their lack of ability to describe decision making or other thought and learning processes occurring in the minds and practices of physicians.

Although quantitative methods demonstrate usefulness and form a major part of the biomedical research model, they often do not describe the complexity of change and the change process. Here, qualitative methods such as focus groups, interviews, chart-stimulated recall,¹⁷ and other methods may generate rich and explanatory data. Further, such study methods may themselves be subject to a form of metaanalysis.¹⁸ Finally, we strongly recommend that future research undertakes cost-effectiveness and cost-benefit analyses.

High-priority topics include clinical areas where there is a documented gap between best clinical evidence and current practice as well as areas of educational research need. The latter will require the development of strategies for further identifying the variables and prioritizing the gaps in our knowledge about CME.

Dr. Davis has received grants from the University of Toronto for the redevelopment of a database and $100,000 in grants from the Ministry of Health, Ontario, Canada.

Dr. Bordage has no conflicts of interest to disclose.

COL Moores has no conflicts to disclose.

Dr. Bennett has served as a consultant with Reed Medical Education about CME; amount is under $2,000.

Dr. Marinopoulos received salary support from the Johns Hopkins EPC and the Agency for Healthcare Research and Quality for his role as co-principal investigator of the review of the effectiveness of CME. He also participated in the Genentech Independent Medical Education Advisory Board following the publication of the Agency for Healthcare Research and Quality Evidence Report.

Dr. Mazmanian has no conflicts of interest to disclose.

Dr. Dorman has no relevant financial conflicts to disclose. He previously served as a consultant to an incubator company, Electrocare, Inc., and once owned stock in Visicu, Inc., since bought by Phillips.

Dr. McCrory received salary support through his academic institution from both federal contracts and contracts with not-for-profit organizations since this report was completed. He has undertaken two consulting relationships unrelated to this report, including serving on a data-safety monitoring committee for a new device trial and as an expert witness in a lawsuit against a product made by Pfizer.