Continuing Medical Education Effect on Physician Knowledge Application and Psychomotor Skills: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines FREE TO VIEW

Despite strong evidence that didactic lectures and unsolicited, mailed, printed material do not produce physician behavioral change or improve patient care,^1–4 they remain popular methods for providing continuing medical education (CME).^1,5 As a result, there is strong pressure on several fronts to reform CME to a process that produces physician change and improves patient care.^1,6 There is a clear need for guidance regarding best practices to inform both providers and consumers of CME so that these programs can change in ways that improve patient outcomes. As an example, psychomotor skills training is critically important in modern medicine. More than two thirds of a medical career is spent in a post-graduate medical education (GME) environments,¹ but recent developments in laparoscopic and minimally invasive surgery (both with significant learning curves) call for training large numbers of physicians who are no longer in GME programs.^7–9 Although limited research on psychomotor training in GME exists and is evolving with the increased use of simulation and virtual reality training,^10–14 it is not clear that these data apply to physicians outside of GME. There is scant information on the effectiveness of CME for teaching psychomotor skills⁸ or guidance for introducing new techniques and procedures to the post-GME physician in a way that improves patient outcomes. This article reviews the available literature and provides recommendations for use of CME in improving physician application of knowledge and teaching psychomotor skills.

The guideline panel reviewed evidence tables and a comprehensive review of the effectiveness of CME developed by The Johns Hopkins Evidence-based Practice Center at the request of the Agency for Healthcare Research and Quality (AHRQ) [AHRQ Evidence Report].¹⁵ The processes for developing the comprehensive review are listed in the Methods article.^15a For this section, the guideline panel considered studies relevant to the effect of CME on physician knowledge application and psychomotor skill development. In the AHRQ Evidence Report,¹⁵ the term skills was used to refer to both psychomotor skills, such as joint injection or the performance of a physical examination, and cognitive skills, such as assessment of depression, the application of patient management, and the use of critical appraisal skills for assessing the medical literature. The panel chose to substitute the term knowledge application in place of cognitive skills to better differentiate this from knowledge acquisition, which is addressed in the Physician Knowledge article.^17a In addition, application of communication techniques was considered a cognitive skill in the evidence report, and studies addressing this are considered with knowledge application in this article. From the 136 studies reviewed in the AHRQ Evidence Report, those addressing the impact of CME on knowledge application or psychomotor skills were identified and retrieved for review. The recommendations listed herein were developed using the American College of Chest Physicians guideline grading system, which is outlined in detail in the Methods article.^15a

Overall, the data for making decisions related to CME and physician knowledge application are scant. The structured review¹⁵ identified only 15 articles^16–30 that addressed either knowledge application or psychomotor skills training. Many of these articles had significant methodological flaws, and the overall quality of evidence is low.

Knowledge application (cognitive and communication skills) was addressed in 12 articles^{16–22,24,26,27,29,30} and included the ability to calculate correct medication doses for pediatric patients, communication skills, diagnostic accuracy for psychiatric patients, critical appraisal skills for assessing medical literature and application of evidence in medical care, diagnostic accuracy and evaluation of skin cancer, headache and COPD, application of cancer control skills, and correct utilization of oral antibiotics. These articles studied only primary care providers (family practice, pediatrics, internal medicine). No data were available for specialists or surgeons. Eleven of the 12 studies^{16–22,24,26,29,30} showed improvement in knowledge application over the short term. Only six studies^{17–21,26,30} provided assessments beyond 30 days, and all showed that improvement was maintained. The only study that did not show an improvement in knowledge application was a controlled study²⁷ involving academic internists and intervention with electronic delivery of a weekly structured summary of research selected from core medical journals. After a 3-month intervention, there was no increase in self-reported use of evidence in patient care compared to a control group; however, the internists reported incorporating evidence into 60% of patient care at baseline. The experimental group increased its reading efficiency compared to the controls and indicated that 20% of the selected articles would have been missed without the summaries.

A variety of instructional media were employed in the studies cited previously. Four studies^16,18,24,30 used videos or CD-ROMs, four studies^24,26,29,30 used print media, 4 used computer-based information (e-mail,²⁷ Web-based programs,^20,21 Listservs²⁶). Two studies^17,22 used audio methods, and eight studies^{16–18,22,24,26,29,30} used live media. Eight^{16–18,22,24,26,29,30} of the 12 studies^{16–22,24,26,27,29,30} used multiple media. In all three studies^19,29,30 that compared two or more experimental groups, the same media were used in each intervention. Given the limited data and lack of direct comparison among media, no specific recommendations can be made for optimal media use in CME to effect change in physician knowledge application.

Multiple educational techniques also were employed in the 12 studies that evaluated physician knowledge application. Eight studies^{16–19,22,24,29,30} used lecture, four studies^18,22,29,30 used role playing, five studies^18,24,27,29 used readings, six studies^{17,18,22,26,29,30} used discussion groups, three studies^21,22,30 used feedback, and three studies^17,22,29 used audiotaped patient encounters. Listservs,²⁶ problem-based learning,¹⁹ and case-based learning^16,20,30 were other techniques studied. Only two studies^19,30 directly compared two or more educational techniques, and the one study¹⁹ that demonstrated a difference between techniques had methodological flaws. Eight^{16–18,21,22,24,27,29,30} of the 12 studies used multiple techniques in the study groups, and several studies^26,27,30 used similar techniques in both the control and the study groups. Based on the limited data and lack of direct comparison among methods, no conclusions can be made regarding the preferred educational techniques for optimizing physician knowledge application.

The majority of studies provided multiple exposures to the CME material, often with differing media and educational techniques. Eight studies^{16–19,22,26,29,30} specified available exposure times, although in some,^{20,21,24,26,27} the utilization of CME was incompletely specified. Reported CME exposure times ranged from 2 h¹⁹ to 48 h.²⁶ Three^16,18,19 of the studies specified a single exposure involving a minimum of 2 h of CME for at least one group in the study. Two of the studies with single exposures^16,18 used multiple media and educational techniques in sessions lasting 3.5 h¹⁶ and 8 h,¹⁸ respectively, raising the question about what constitutes a single educational exposure. Only one study compared multiple exposures to a single exposure.¹⁹ In that study, multiple exposures were associated with better outcomes, but there were significant differences in techniques employed and time devoted to CME between the two groups. The groups also were not randomly assigned. Despite the limitations in the data, the guideline panel believed that the most convincing data demonstrating improvement in physician application of knowledge were generated in studies using multiple exposures. The consensus of the panel was that both longer duration and multiple exposures appeared to provide greater benefit in improving physician application of knowledge, although the evidence to allow a recommendation for longer duration was not believed to be sufficient at this time given the available data.

Psychomotor skills were addressed in only three studies^23,25,28 identified in the structured review. Hergenroeder et al²³ investigated two methods for teaching pediatricians to perform ankle and knee physical examinations. Seventy-five pediatricians were randomly assigned to either a group that received videotape and written instruction or a group that received videotape, written instruction, and a skills-building workshop. When tested with standardized patients at 4 to 5 months after training, both groups improved compared to baseline, but the group with hands-on experience in the skills-building session improved to a greater degree.

Rodney and Albers²⁸ compared two methods of teaching primary care providers to perform flexible sigmoidoscopy. They compared a large group-lecture format supplemented with videotapes and 3 h of mannequin practice with a small-group format, less lecture time, a more intensive workshop session, more faculty involvement, and a requirement for the participant to demonstrate proficiency with the sigmoidoscope. The groups were not randomly assigned, and there was a substantial difference in CME time between the two groups. The outcome assessment 12 to 16 months after completing the courses used self-reported data about the learning curve, defined as time to complete a procedure and the depth scoped as a function of the number of procedures performed. The small-group-format study participants reported a shorter procedure time for the first 10 procedures, but no other differences between the two groups were noted. The authors also noted but did not discuss that the small-group-format participants appeared less likely to purchase a sigmoidoscope and performed fewer procedures per provider than the large-group participants. No comparison to a non-CME group was made, although comparison to published complication rates for colonoscopy was made.

Leopold et al²⁵ studied three methods for teaching primary care providers how to perform a knee injection. Ninety-three participants in a CME workshop were randomized to one of three groups after attending a 15-min lecture on knee landmarks and injection techniques. One group received written instruction in knee injection, the second watched a videotape illustrating the procedure, and the third received hands-on instruction with supervised practice and feedback. All three groups improved compared to the baseline assessment, which was a simulated injection using a knee model, and no difference among groups was demonstrated.

In all three studies,^23,25,28 all methods studied provided some evidence for improved psychomotor skills, although data to make specific recommendations are very limited. Only simple outpatient procedures were studied, and all included only primary care providers. All studies provided training using multiple instructional techniques and multiple media. Thus, no specific recommendations regarding preferred modalities, media, frequency, or duration of CME are possible. As a result, no recommendation can be made regarding procedural skills training.

The available data show that CME can be effective in physician knowledge application, and the guideline panel recommended that CME be used for this purpose. With only 15 studies addressing this topic, the data are too limited to allow recommendations regarding optimal media or educational techniques. The available studies employed a variety of media and techniques, often in the same groups, and direct comparisons of competing media or educational techniques are infrequent and often marred by study design flaws. Although issues also existed with the data addressing exposure, most of the positive studies employed multiple exposures and longer duration activities, which is in keeping with previously published reviews on CME effectiveness.³¹ Thus, the guideline panel recommended that CME programs employ multiple exposures to improve physician knowledge application. The data for CME effectiveness in psychomotor skills training include only three studies and are too limited to merit a recommendation.

A better understanding of what works in CME is required if CME is to be effective in improving physician application of knowledge; psychomotor skills; and, ultimately, patient outcomes. Well-designed studies directly comparing different media and educational techniques are needed. Additionally, defining an appropriate duration and number of exposures to improve knowledge and skills is critical for CME in a time when the costs, effectiveness, and sponsorship of this $2 billion-a-year industry is increasingly called into question.⁵ Broadening the study populations to include more than primary care practitioners and additional studies addressing psychomotor skills also are critically needed, and it would be important to learn whether the principles developed for medical school and GME also are applicable to CME, as some^8,32 have suggested. Until such data are available, our recommendations provide the best guidance for CME providers and physicians looking to optimize CME effectiveness with regard to physician knowledge application and serve as a starting point for additional research.

Dr. O'Neil has no conflicts of interest to disclose.

Dr. Addrizzo-Harris has no conflicts of interest to disclose.