Continuing Medical Education Effect on Physician Knowledge: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines FREE TO VIEW

There are important associations between physician knowledge and practice outcomes. For example, a positive association exists between knowledge- based certification examination results in internal medicine and actual clinical performance,¹ with disattenuated correlation coefficients between 0.55 and 0.59. There is also a relationship between the knowledge-based certification status of internists and cardiologists and the mortality rate of their patients following an acute myocardial infarction, with 19% lower mortality rates for the patients of certified specialists.² Patients of surgeons certified by the American Board of Surgery had lower mortality and morbidity rates following segmental colon resection than those of physicians who were not certified.³ Similar improvement in results related to board certification status have been found in studies of obstetrical care,⁴ surgical mortality,⁵ outcomes after abdominal aortic aneurysm rupture,⁶ malpractice claims,⁷ and disciplinary actions.⁸ Thus, assessing physician knowledge immediately or some time after attending a continuing medical education (CME) activity is a legitimate endeavor. Knowledge acquisition and retention represent two of the many outcomes worthy of assessment in CME, along with participant satisfaction; transfer of skills into practice; and ultimately, patient and population outcomes.

In CME, the assessment of physician knowledge in the context of clinical practice can serve the following two main purposes: to assess learning needs; and to assess physicians' ability to apply their knowledge in explaining and managing clinical problems. The assessment of the knowledge needs of practicing physicians, both perceived and nonperceived, constitutes one of the elements used to determine the content and format of CME interventions. The importance of educational needs is highlighted in the Accreditation Council for Continuing Medical Education accreditation process.⁹ The chain of outcomes in CME accreditation criteria goes from physician competence to physician performance and patient outcomes (compliance criterion No. 3). Viewed in the context of the pyramid of Miller¹⁰ for assessing clinical competence, the lowest level of assessment is factual knowledge (knows), followed by competence or applied knowledge (knows how), performance (shows how), and action (does) [Fig 1]. The real test of competence (knows how) is in the clinician's ability to understand underlying concepts and principles, to think through problems, and to make decisions and explain findings and mechanisms, the second level in the Miller pyramid.¹⁰

Four key questions arise from CME activities that offer knowledge objectives. Do these CME activities improve physician knowledge? Are multimedia interventions (eg, live, computer based, Internet based, use of video, audio, or print) in CME preferable to single-medium interventions to improve physician knowledge? Are multiple instructional techniques (eg, academic detailing, case-based learning, demonstrations, discussion groups, lectures, mentoring, readings, or simulations) preferable to single-technique CME interventions to improve physician knowledge? Are multiple exposures to CME content preferable to a single exposure to improve physician knowledge? A systematic review was undertaken to answer these questions.

Implications for the physician-learner from this portion of the review suggest that participating in CME activities that do not have multiple methods of learning will have a minimal impact. CME activities that require the application of knowledge are a strong indicator of expert learning and competency. Physician-learners have better knowledge gain when asked to gather and analyze history and physical examination findings presented on paper or through a simulated or live patient, make a diagnosis, explain the underlying pathophysiology, and make recommendations on how best to manage the situation. From the physician-teacher perspective, a CME activity should be designed to include multiple educational media and techniques, and, where possible, multiple exposures.

As detailed in the “Methods” article^10a, the Johns Hopkins Evidence-based Practice Center conducted a systematic review of the effectiveness of CME.¹¹ Of the 136 studies in the systematic review, 39 studies ^12–50 addressed a total of 41 knowledge objectives. Of these 39 studies, 11 studies^{20,28,30–32,34,35,37,44,46,50} were excluded from further analyses because, although they used a comparison group, they did not have a control group; thus, the effectiveness of the intervention compared to a nonintervention, control-group baseline could not be determined. Consequently, 28 studies with an adequate control group were included in the analyses, addressing a total of 29 knowledge objectives. The data from each study were classified as to whether the knowledge objectives were met, were not met, or resulted in mixed results (see Methods article^10a for definitions).¹¹ Thus, the knowledge gain was measured using the control group as the base level. To assess the duration of knowledge retention, the extent of time between the CME activity and the test of knowledge was classified as short term (< 30 days), long term (≥ 30 days), or not clearly reported. The heterogeneity of the content, the designs of the studies, and the weaknesses in the study designs warranted an overall grading of C (low) for the quality of the evidence by the guideline panel, using the American College of Chest Physicians grading system.⁵¹

Twenty-two of the 28 studies (79%)^{12–14,17–19,21,24,26,27,29,33,36,38,39,41–43,45,47–49} showed improvements in knowledge. Four studies^16,22,23,25 (14%) failed to show improvements in knowledge, and two studies^15,40 (7%) had mixed results. None showed a decrement in knowledge.

The results from 19 of the 28 studies came from long-term assessment of knowledge retention, as follows: 15 studies (of 22) from the improvement group, 3 studies (of 4) from the no-improvement group, and 1 study (of 2) from the mixed-results group. The knowledge retention from the remaining nine studies was too sparse or heterogeneous to conduct a meaningful analysis.

In summary, most studies (79%) showed that CME activities were associated with improvements in knowledge. The majority (15 of 22 studies; 68%) showed long-term knowledge retention.

The CME activities from the 136 studies included in the Agency for Healthcare Research and Quality Evidence report¹¹ were classified according to the following three facets of CME activities: the instructional media, the instructional techniques, and the frequency of exposure to a given CME program. Overall, multimedia (eg, video, audio, or print), multiple instructional techniques (eg, discussion groups or case-based learning), and multiple exposures (sessions) were used most often. (Methods article^10a, Tables 3 and 4 provide detailed description of the media, technique, and exposure characteristics).¹¹

Instructional media were classified according to eight different types (Methods article^10a, Table 3).¹¹ Studies also were classified as having used a single-medium intervention or multimedia intervention or as comparative studies of single vs multimedia interventions.

Four of the 28 studies included in the analyses compared single-medium vs multimedia CME interventions. Of these, three studies^13,38,43 showed that multimedia interventions had a greater benefit than single-medium interventions, including a print-based, single-medium intervention group.

Of the nine studies using a single-medium intervention, seven^{21,26,29,36,45,47,49} (78%) showed improvements in knowledge. Of the 15 studies using multimedia interventions, 12 studies^{12,14,17–19,24,27,33,39,41,42,48} (80%) showed improvements in knowledge.

In the four studies that did not show improvements in knowledge, one²³ used a single-print medium, whereas the other three^16,23,25 used multimedia interventions (live, audio, and print; live and print; live intervention vs live Internet with non-real-time reading material). In the two studies with mixed results, one⁴⁰ used a single-medium intervention (live), and one¹⁵ used a multimedia intervention (live Internet with printed material).

The overall effectiveness of both single-medium and multimedia interventions was similar. However, more studies evaluated multimedia formats, and three of the four comparative studies favored multimedia intervention.

Instructional techniques were classified according to 17 different types (Methods article^10a, Table 4).¹¹ Studies also were classified as having used a single-technique intervention or multiple-technique intervention or as comparative studies of single vs multiple technique interventions.

Thirty studies with an adequate control group addressed this question, with a total of 31 knowledge objectives. Two studies^28,30 not previously used were added for this question because they contained a concurrent comparison group that used a different instructional technique. Overall, 22 studies (73%) showed improvements in knowledge, including 2 with single-technique interventions and 20 with multiple-technique interventions. Six studies did not show improvements, and 2 contained mixed results.

Five of the 30 studies compared single-technique vs multiple-technique CME interventions. Of these five comparative studies, two^29,38 showed that multiple-technique interventions had a greater benefit on knowledge improvement than single-technique interventions, and one²¹ showed the opposite, favoring a single-technique problem-based learning intervention. Of the two studies using single-technique interventions, one¹⁷ showed improvements in knowledge, and one⁴⁶ did not. Of the 23 studies using multiple techniques, 18 studies^{12–14,18,19,24,26,27,29,33,36,39,41,42,45,47–49} (78%) showed improvements in knowledge.

Each of the 22 studies showing improvements in knowledge used a variety of combinations of instructional techniques, including, for example, case-based learning most commonly combined with discussion group and readings; lecture with readings, standardized patient, or team-based learning; or readings and discussion group. In the six studies that did not show improvements in knowledge, only one²³ used a single-technique intervention (reading). The remaining five studies^{16,22,25,28,30} used multiple-technique interventions that combined two to four of the following instructional techniques (ie, three interventions with two techniques, one with three techniques, and two with four techniques): case-based readings, discussion group, feedback, lecture, point of care, readings, and role play. The other two studies^15,40 reporting mixed results used multiple-technique interventions (case-based learning with discussion group and lecture with discussion group). Finally, of the studies for which the time of knowledge assessment (short or long term) could be clearly established, the majority (88%) were associated with long-term assessment of multiple-technique interventions.

Given the heterogeneity of combinations of instructional techniques used in the studies reviewed and the array of results reported across techniques, no firm conclusions can be made. However, because a majority of the studies showing improvement in physician knowledge used more than one technique, the Johns Hopkins Evidence-based Practice Center¹¹ concluded that “multiple techniques that most commonly include case-based learning seem to be more associated with improvements in knowledge.”

The frequency of exposure to CME content in a given CME program (number of CME sessions) was classified in each study according to the following three categories: single exposure (once); multiple exposures (multiple times); and single vs multiple exposures. For this question, 27 studies with an adequate control group addressed this question, with 28 knowledge objectives. One study⁴³ was excluded because the description of the exposure frequency was not sufficiently clear to classify. Overall, 21 of the 27 studies (78%) showed improvements in knowledge as follows: 5 studies^{12,13,26,36,49} (of 5) from single-exposure CME interventions; 16 and from multiple-exposure interventions (ie, 12 [of 17] from multiple-exposure studies and 4 [of 5] from comparative [single vs multiple] studies). Four studies did not show improvements in knowledge, and two reported mixed results.

The five studies^{12,13,26,36,49} of single-exposure CME interventions all led to improvements in knowledge, four with long-term assessments of knowledge and one with the time of knowledge assessment not clearly specified. The four comparative studies^19,21,24,38 with improvements in knowledge used multiple-exposure interventions (vs single exposure), three with long-term assessments of knowledge and one with the time of knowledge assessment not clearly specified. Of the 17 multiple-exposure studies, 12^{14,17,18,27,29,33,39,41,42,45,47,48} showed improvements in knowledge, with knowledge assessed at different times (1 with short-term assessment, 8 with long-term assessments, and 3 with the time of assessment not clearly specified).

In the four studies^16,22,23,25 with no improvements in knowledge, 0 came from single-exposure studies, and one¹⁶ came from a comparative study with long-term assessment of knowledge. Three studies^22,23,25 came from multiple-exposure studies, two^22,23 with long-term knowledge assessment and one²⁵ with the time of assessment not clearly specified.

In summary, despite the heterogeneity of the studies reviewed and the fact that all five studies that used a single-exposure CME intervention showed improvements in knowledge, the other results, with head-to-head comparisons, “imply that when possible multiple exposures [to CME content] produces better knowledge gains.”¹¹

The overall quality of the evidence from the studies reviewed to assess the impact of CME interventions on short-term and long-term knowledge acquisition and retention was low. In addition, the heterogeneity of the studies in terms of study designs, content areas, educational methods, and frequency of exposure preclude firm conclusions. Nevertheless, the results from the majority (79%) of those studies showed that CME activities were associated with improvements in knowledge, with both short-term and long-term gains, thus warranting a recommendation to use CME activities to improve physician knowledge.

The type of knowledge assessed was not reviewed in detail in this study but warrants further comments. Some educators divide knowledge into the following two broad categories: factual knowledge (to define, describe, list, name, recall) and applied knowledge, mirroring the “knows” and “knows how” in Miller's pyramid of clinical competence.^10,52 Applied knowledge calls on greater cognitive skill, as follows from the taxonomy of mental skills by Krathwohl et al⁵³: comprehension (to explain), application (to solve), analysis (to compare and contrast), synthesis (to create, summarize), and evaluation (to justify, defend). Rote memorization of facts is a poor predictor of problem-solving ability, whereas applied knowledge best predicts deeper learning and expert performance.^54–56 It was the French scientist Poincaré⁵⁷ who drew the following analogy to distinguish facts from applied knowledge: “Science is built up with facts, as a house is with stones. But a collection of facts is no more a science than a heap of stones is a house.” National testing agencies, such as the United States Medical Licensing Examination, recognize the importance of applied knowledge and require that their test items assess applied knowledge related to key concepts and principles essential for clinicians to understand.⁵² This is the difference, for example, between asking a clinician to describe Korsakoff syndrome and asking a clinician to gather and analyze the history and physical examination findings from a postoperative patient who is agitated (presented on paper or through a simulated or live patient), make a diagnosis, explain its mechanism, and manage the situation. If one cannot apply what one knows, then that knowledge is not useful. Implied in the Accreditation Council for Continuing Medical Education accreditation criteria⁹ is the assessment not only of knowledge, factual or applied, as part of the needs assessment process, but also of applied knowledge (competence) as an outcome measure of physician competence. The assessment of applied knowledge with written or performance tests is useful to evaluate that dimension of CME outcomes because it predicts actual performance in practice.^1–3,58

Although it is of low quality, the evidence concerning the use of media and instructional techniques and the frequency of exposure suggests that multimedia, multiple instructional techniques, and multiple exposures be used whenever possible in CME. Although all three sets of results point to the conclusion that multiple modalities are most effective, reality actually is more complex. The interaction of each variation of each dimension is an important consideration for CME providers, teachers, and attendees. For example, CME providers must address multiple audiences with different needs and preferences. On the other hand, attendees may reject a single format because it does not fit their preferences. No single solution to these issues is possible; rather, the instructional technique should be chosen to best engage the attendees (eg, group discussions, demonstrations) while the medium helps to present the content (eg, printed materials, video presentations). Academic detailing is a good example of the importance of considering instructional media and instructional techniques as synergistic. Some physicians may prefer demonstrations with printed materials, whereas others may prefer group discussions with videos. Some teachers may be better at leading groups and others at mentoring. The frequency of exposure further increases the possible interactions among these variables. Elucidating what works to improve an individual physician's competence and performance involves the consideration of complex interactions among intervention characteristics, physician characteristics, practice setting characteristics, and patient characteristics.^59,60

Physician-learners progress at their own rates, depending on motivation, knowledge of a problem, or the perception of a gap between their current knowledge and skills and those needed.⁶¹ Knowledge is necessary but not an end in itself that will lead to a change in physician behavior or patient outcomes.⁶² When barriers to change are addressed or gaps are demonstrated and resources deployed to help the learner, change may be expected to occur.^62,63 Consistent with prior studies, combinations of instructional techniques, with interactive components such as case-based discussions or role playing, were shown to effect change in knowledge.

These interactions and the variety of possible combinations, while beneficial for learners, create major study design problems and potential confounders when conducting research on the effectiveness of any single dimension or sets of dimensions. Was knowledge gained because of multimedia interventions or single vs multiple exposures or because multimedia may simply represent multiple exposures to the same content? Cook⁶⁴ discussed these design issues in the context of conducting research on computer-based instruction. Using a slightly different taxonomy, he identified the following four dimensions of educational interventions that need to be controlled across experimental and control groups when comparing different instructional interventions: the medium (eg, classroom, audiotapes, Web based), the configuration (eg, discussion boards, tutorials, simulations), the instructional method (eg, practice, feedback), and the presentation (eg, color, sound, fidelity). To understand the unique contribution of each dimension (or specific sets of dimensions), the researcher must vary only one dimension (or set) at a time, holding the other dimensions constant. Comparisons must be made within dimensions and not between dimensions to avoid confounding variables. Even a perfectly designed and executed randomized clinical trial is still subject to confounding variables if the interventions being compared differ in multiple ways. Two of the main problems encountered in the studies reviewed were the heterogeneity of the dimensions across studies and the presence of confounding variables. A more systematic approach is needed to better assess the unique and combined contributions of media, techniques, and frequency of exposure and their impact on outcomes, including physician knowledge and performance and patient outcomes following CME interventions. Much of the research conducted in CME, and medical education in general, is done in isolation and on an opportunistic basis because of either available funding or an investigator's passing interest. Programmatic and collaborative research is needed to better assess the unique and combined contributions of the various dimensions of CME,⁶⁵ such as media, techniques, and frequency of exposure.

Dr. Bordage has no conflicts of interest.

Dr. Carlin has no conflicts of interest.

Dr. Mazmanian has no conflicts of interest.