Continuing Medical Education Effect on Practice Performance: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines FREE TO VIEW

There has been considerable debate and widespread skepticism about the effect of continuing medical education (CME) on the performance of physicians in the practice setting.^1,2 The relationship between education and performance is of great importance to health service managers, quality improvement coordinators, guideline implementers, licensing bodies, policymakers, practitioners, and others. In addition, the suggestion that CME can and should lead to change in clinical performance is an increasingly important principle of the accreditation process used by the Accreditation Council for Continuing Medical Education and to systems of relicensure, recertification, and many other stakeholders in the US healthcare system.

Implications to the physician-learner from the literature review in this article suggest similar findings to those in the preceding two articles. The primary focus here, however, is on physician practice performance as opposed to knowledge acquisition or application. Here again, the guideline panel suggests that live, face-to-face educational activities are effective, especially when combined with multiple exposures to the information following the live educational activity. To the physician-teacher and those who depend on findings from CME activities, the type and amount of multiple exposures are still speculative at best. This article also emphasizes the need for additional studies to be conducted by physician-teachers to compare the number of exposures following a live education activity that are most effective in improving physician performance.

The guideline panel reviewed the evidence tables and comprehensive review of the effectiveness of CME developed by The Johns Hopkins Evidence-based Practice Center (EPC), which is detailed in the methods article.^2a For this article, we focused on the effect of CME on physician practice performance. These effects were measured in either the short term (< 30 days after an intervention) or the long term (≥ 30 days postintervention). The panel noted several limitations in the methods of the EPC review, which are detailed in the “Science of Continuing Medical Education”^2b article of this supplement and at the conclusion of this article.

The review identified 105 studies, which evaluated the impact of CME on short- and long-term physician practice performance. The majority (61 studies, 58%) met practice objectives. A wide mix of objectives was studied, including prescribing; screening; counseling about smoking cessation, diet, and sexual practices; guideline adherence; and others. Fifty studies^3–51 were successful in meeting many objectives and demonstrated long-term effectiveness. Among these, evaluation duration ranged from 30 days to 6 months (17 studies) to 1 year or longer (30 studies).

The majority of studies reported positive outcomes, but slightly less than 30% did not. Of these, 24^{7,18,27,29,33,34,52–69} analyzed their outcomes for > 30 days, 2^70,71 had shorter evaluation periods; and 3^72–74 did not report their duration of evaluation. Nine studies^{58,59,75–81} showed mixed results, whereas one¹⁸ was unclear about whether it met objectives. Fourteen studies^82–95 lacked a control group; thus, no reliable measurement of effect could be made.

We posed several questions in this process. Is there a difference in the effect of the use of single vs multiple media? What is the effect of the use of a specific media in CME on practice performance? Are some media better or worse than others in changing performance? Twenty studies used single live media of which one half met their performance objectives. Nine of these studies^{3,6,23,35,39,45,51,107,108} reported long-term performance outcomes, suggesting that the use of single live media methods can generate long-term effects on practice performance objectives;one⁹⁷ study assessed outcome < 30 days and met its objectives. Three studies^75,79,80 using single live events reported mixed results, and three^60,70,72 did not change practice behavior. Finally, four^85,86,90,91 studies lacked a control group.

Single print media, particularly comprehensive, unsolicited materials, are not effective in the short- or long-term achievement of practice performance objectives. Nine studies examined the impact of single print media. One⁹⁸ met objectives but did not report evaluation duration, whereas four^52,55,63,66 did not meet objectives over the long term. Four further studies did not meet objectives, with two^71,74 not specifying an evaluation period and two^93,95 lacking a control group. Data available on the use of other single media are scant. One study⁶⁸ using an Internet medium reported a long-term evaluation, but it was unclear whether it met objectives. Three studies either did not meet objectives,⁷³ lacked a control group,^82,87 or did not report the methods used.⁸⁷

Of 57 studies using multiple media, the majority (40 studies) met their objective. Of these, 31^{4,7–10,12–20,22,24,31–34,36–38,42–44,46,48–50,99} included long-term evaluations, suggesting that multiple media methods can have a favorable long-term effect on practice behaviors. Nine studies^96,100–107 met objectives using multiple media but did not specify the timing of evaluation. In contrast, 4 studies^59,76–78 showed mixed results over the long term, 14^{7,18,33,34,53,54,56,57,59,61,64,65,67,69} did not achieve their objectives, and 3^83,88,92 did not include a control group. Overall, most studies suggested that CME activities that use multiple media have both a short- and a long-term effect on practice behavior objectives.

Fifteen studies provided an opportunity for direct comparisons of single and multiple media methods. Of these, 10^{5,21,25,27–30,41,46,47} met 11 objectives, and all were evaluated over the long term. These studies suggest that both single and multiple media methods have a positive short- and long-term effect on practice behavior objectives, although many were methodologically flawed and did not permit direct comparisons or a recommendation.

The EPC review also focused on the use and effects of specific educational techniques employed within the context of a single medium, such as a live activity. What is the effect of specific CME educational techniques on practice performance? Is there a difference in outcome when one technique is used compared to instances in which multiple techniques are used? The techniques studied included academic detailing, audience response systems, case-based learning, clinical experiences, demonstrations, discussion groups, feedback, lectures, mentoring or precepting programs, point-of-care techniques, problem-based learning, team-based learning, programmed learning, readings, role play, simulations with standardized patients, and writing.

Eleven studies evaluated the impact of a single technique on practice performance. Three of these studies^14,97,98 displayed some evidence of positive changes, although only one¹⁴ met objectives in the long term. In contrast, 4 studies^52,64,71,73 did not meet objectives using a single technique, 1⁸⁰ yielded mixed results, and 3^85,91,95 lacked a control group. In summary, these studies suggest that a short- or long-term positive effect on practice behavior objectives is not clearly achieved through a single technique.

A total of 76 studies with 98 objectives evaluated the short- and long-term impact of multiple techniques on practice behavior objectives. Of this number, more than half (47 studies) achieved their objectives. Of these 47, 39^{4–10,12,13,15–21,23,24,26,31–34,36–39,42–51,108} measured their effects over the long term. A further eight^{96,100,101,103–107} used multiple techniques but did not report the timing of evaluation. In contrast, 16 studies^{7,18,33,53–57,59–61,65–69} evaluated long-term outcomes but did not meet objectives, and 3^70,72,74 that did not specify evaluation timing failed to meet objectives. Fifteen studies^{59,75–79,83,84,87,89,90,92,93,109,110} did not permit interpretation; they were methodologically flawed, did not describe evaluation timing, lacked a control group, or demonstrated mixed results.

Eighteen studies^{3,25,27–30,35,40,41,58,62,63,81,82,86,94,99,102} compared the use of single and multiple educational techniques in CME. Ten^{3,25,27–30,35,40,41,99} of these met their objectives over the long term and suggested that multiple techniques may have a more advantageous short- and long-term effect on practice behavior objectives. Two studies^58,81 reported mixed results, whereas an additional four^29,58,62,63 either lacked a control group or did not meet objectives. Two^82,94 studies were methodologically flawed. In summary, the evidence from these studies in which direct comparisons were possible indicates that CME activities that use multiple educational techniques may have a greater overall positive short-term effect, long-term effect, or both on practice behavior objectives than those that use only a single technique.

Thirty-seven studies evaluated the impact of single exposure to the CME activity on practice performance. Of these, just under half (18 studies)^{4,5,9,21,23,25–27,38,39,41,43,45,46,51,100,101,108} demonstrated a positive effect. In contrast, 8 studies^{27,52,61,62,64,69–71} did not meet their objectives, 5^{85,86,88,90,91} lacked a control group, and 6^{75,76,78–81} showed mixed results. Therefore, a single CME exposure may have a positive short- and long-term effect on practice behavior objectives.

The impact of multiple exposures to the CME activity was evaluated in 55 studies encompassing 72 objectives. In this set of studies, almost two thirds (35 studies) met their objectives. Of these, 30 studies^{3,6,8,10,12–20,22,24,28,31–36,40,42,44,47–50,99} continued to meet their objectives over the long term. A further five studies^{96,98,103,104,107} met objectives but did not report evaluation timing, and two studies^58,59 displayed mixed results. In contrast, one third (18 studies) did not achieve an impact on performance. Two of these studies^72,73 did not report evaluation duration, whereas 16^{18,33,34,53,55–60,63,65–68,111} with long-term evaluation periods did not realize their objectives. Six studies^{82,85,86,88,90,91} lacked a control group.

Eight studies^{7,29,30,77,83,92,94,105} offered direct comparisons between single and multiple exposures; however, they lacked control groups, showed mixed results, or both. Thus, no strong conclusions are possible from this portion of the EPC review. Overall, the data suggest that multiple exposures to CME activities may have more positive effects than single exposures on practice performance in both the short and the long term.

It has long been accepted that CME is an important educational intervention for the improvement of practice performance in health care, but the evidence from previous systematic reviews and other data offers less than strong support to this belief.^1,2,112 Direct comparisons between the EPC review and these studies is made problematic by the EPC's inclusion of self-reported performance outcomes in some studies. This use of self-report with its potential for bias is in contrast to other systematic reviews, which limited the outcomes to objective measures of performance change only.

Nevertheless, the guideline panel accepted the robustness of the EPC report that suggests a more positive interpretation of the effect of CME, noting several themes in the context of CME's role in improving or maintaining practice performance. First, in the broad construct of CME as a live, face-to-face activity, evidence indicates that this form of education may be effective, especially in settings that permit multiple exposures. Second, the review shows improved results with the use of multiple media and multiple educational techniques; this finding is entirely consistent with principles of adult learning theory.^113,114 Third, it is clear that much useful evidence is still missing in terms of related research questions and the more routine inclusion of measures of practice performance in CME activities. Among the missing data is sufficient comparative evidence on which to derive conclusions about such issues as the type and nature of media, the type and nature of educational techniques (eg, case-based learning, interactivity, sequencing of multiple activities), the confounders of learner motivation and setting, and the degree of change required. These questions and a further discussion about possible limitations to the review are framed in the mandate for establishing a robust research agenda, as discussed in “The Science of Continuing Medical Education” article in this supplement.^2b

Dr. Davis has received grants from the university of Toronto for the redevelopment of a database and $100,00 in grants from the Ministry of Health, Ontario, Canada.

Dr. Galbraith has no conflicts of interest to disclose.