0
Original Research: Education, Research, and Quality Improvement |

Development of a Novel, Multilayered Presentation Format for Clinical Practice GuidelinesPresentation Format for Clinical Guidelines FREE TO VIEW

Annette Kristiansen, MD; Linn Brandt, MD; Pablo Alonso-Coello, MD, PhD; Thomas Agoritsas, MD; Elie A. Akl, MD, PhD, MPH; Tara Conboy, RGN, MSc; Mahmoud Elbarbary, MD, PhD; Mazen Ferwana, MD, PhD; Wedad Medani, MSc; Mohammad Hassan Murad, MD, MPH; David Rigau, MD; Sarah Rosenbaum, PhD; Frederick A. Spencer, MD; Shaun Treweek, PhD; Gordon Guyatt, MD, FCCP; Per Olav Vandvik, MD, PhD
Author and Funding Information

From the Department of Internal Medicine (Drs Kristiansen, Brandt, and Vandvik), Innlandet Hospital Trust, Gjøvik, Norway; Institute for Health and Society (Drs Kristiansen, Brandt, and Vandvik), Faculty of Medicine, University of Oslo, Oslo, Norway; Department of Internal Medicine (Drs Kristiansen and Brandt), Diakonhjemmet Hospital, Oslo, Norway; Iberoamerican Cochrane Centre, Clinical Epidemiology and Public Health Department (Drs Alonso-Coello and Rigau), Institute of Biomedical Research, Sant Pau, Spain; Department of Clinical Epidemiology and Biostatistics (Drs Agoritsas and Guyatt) and Department of Medicine (Dr Spencer), McMaster University, Hamilton, ON, Canada; Department of Internal Medicine (Dr Akl), American University of Beirut, Beirut, Lebanon; National & Gulf Center for Evidence Based Health Practice (Mss Conboy and Medani and Drs Elbarbary and Ferwana), King Saud Bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia; Mayo Clinic (Dr Murad), Rochester, MN; The Norwegian Knowledge Centre for the Health Services (Drs Rosenbaum and Vandvik), Oslo, Norway; and Health Services Research Unit (Prof Treweek), University of Aberdeen, Aberdeen, Scotland.

CORRESPONDENCE TO: Pablo Alonso-Coello, MD, PhD, Iberoamerican Cochrane Centre, Clinical Epidemiology and Public Health Department, Institute of Biomedical Research, Sant Pau, Spain 08025; e-mail: PAlonso@santpau.cat


Portions of this study have been presented in a poster at the 9th Guideline International Network Conference, August 22-25, 2012, Berlin, Germany; the 10th Guideline International Network Conference, August 18-21, 2013, San Francisco, CA; and at the 11th Guideline International Network Conference, August 20-23, 2014, Melbourne, Australia.

FUNDING/SUPPORT: This project has received funding from the European Union’s Seventh Framework Programme for research, technological development, and demonstration under grant agreement no. 258583. The Innlandet Hospital Trust and South-Eastern Norway Regional Health Authority have provided research grants for the MAGIC program.

Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details.


Chest. 2015;147(3):754-763. doi:10.1378/chest.14-1366
Text Size: A A A
Published online

BACKGROUND:  Bridging the gap between clinical research and everyday health-care practice requires effective communication strategies. To address current shortcomings in conveying practice recommendations and supporting evidence, we are creating and testing presentation formats for clinical practice guidelines (CPGs).

METHODS:  We carried out multiple cycles of brainstorming and sketching, developing a prototype. Physicians participating in the user testing viewed CPG formats linked to clinical scenarios and engaged in semistructured interviews applying a think-aloud method for exploring important aspects of user experience.

RESULTS:  We developed a multilayered presentation format that allows clinicians to successively view more in-depth information. Starting with the recommendations, clinicians can, on demand, access a rationale and a key information section containing statements on quality of the evidence, balance between desirable and undesirable consequences, values and preferences, and resource considerations. We collected feedback from 27 stakeholders and performed user testing with 47 practicing physicians from six countries. Advisory group feedback and user testing of the first version revealed problems with conceptual understanding of underlying CPG methodology, as well as difficulties with the complexity of the layout and content. Extensive revisions made before the second round of user testing resulted in most participants expressing overall satisfaction with the final presentation format.

CONCLUSIONS:  We have developed an electronic, multilayered, CPG format that enhances the usability of CPGs for frontline clinicians. We have implemented the format in electronic guideline tools that guideline organizations can now use when authoring and publishing their guidelines.

Figures in this Article

Health-care professionals’ complex informational needs face constraints of time, multitasking, interruptions, and uncertainty.1 To ensure successful dissemination and implementation, clinical practice guidelines (CPGs) must be easy to find, understand, and apply at the point of care26—attributes frequently absent in current CPGs.710 Investigators have begun to address the issue of presentation2,3,1114 in particular strategies for improved risk communication.2,1520 Nevertheless, evidence to guide effective ways of communicating recommendations remains limited.

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group has developed a widely adopted framework for the development of evidence-based CPGs.21,22 In 2011, the GRADE working group, realizing the limited evidence addressing optimal presentation formats, launched the Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE) project, a multinational research endeavor funded by the European Union.23 The Making Grade the Irresistible Choice (MAGIC) research and innovation program was established in 2010 to facilitate the authoring, dissemination, and updating of trustworthy CPGs.24,25 This article describes a collaborative effort of the DECIDE project and MAGIC program to develop a transparent, understandable, easy-to-use presentation format for CPGs.

The previously published23 DECIDE methodology includes three phases: (1) strategy development and user testing, (2) evaluation, and (3) testing with real CPGs (Fig 1). This report focuses on the first phase in which we have developed a CPG presentation format through an iterative process of brainstorming and sketching, advisory group feedback, and user testing. We performed user testing with practicing physicians as representatives of health-care professionals that frequently use CPGs.

Figure Jump LinkFigure 1 –  Development method of the multilayered presentation format. RCT = randomized controlled trial.Grahic Jump Location
Selection of CPG Content to Develop the Presentation Formats

We selected recommendations from published CPGs developed using the GRADE framework26 that were relevant both for primary and secondary care physicians and restructured the content into multilayered presentation formats (Figs 2, 3). For the first round of user testing, we used recommendations addressing aspirin use for the primary prevention of cardiovascular disease and use of anticoagulant therapy in atrial fibrillation.27,28 For the second round, we revised the scenario on atrial fibrillation and added a recommendation for metformin treatment in type 2 diabetes.29 For each recommendation, we developed a clinical scenario introducing a decision about therapy (eg, “Should a 68-year-old man with atrial fibrillation and CHADS2 score of 1 use anticoagulation?”).

Figure Jump LinkFigure 2 –  Presentation format tested in round 1. A, First layer with list of recommendations. B, Second layer with key information. C, Rationale.Grahic Jump Location
Figure Jump LinkFigure 3 –  Presentation format tested in round 2. A, First layer with list of recommendations. B, Second layer with key information.Grahic Jump Location
Brainstorming and Sketching of a Prototype Presentation Format
Investigators:

The research group included frontline clinicians, CPG developers, clinical epidemiologists, and interface designers with a broad international representation.

Process:

A workshop at the start-up meeting of the DECIDE project in 2011 generated a range of new ideas for CPG formats. An iterative process of brainstorming and sketching continued through face-to-face meetings, teleconferences, and e-mail exchanges within the research group.

We used sketching as a way of exploring promising ideas emerging from brainstorming, with the sketches functioning as a prototype. We followed a “mobile first” design approach30 and tailored the first prototype (Fig 2) to a smartphone screen, as this screen is the smallest platform for guideline viewing and represents the most challenging presentation format. We used Blueprint31 (Groosoft.com) to create a mock-up of a functioning CPG on a tablet computer.

Advisory Group Feedback
Participants:

We invited experts on guideline development, design, clinical epidemiology, implementation science, communication, and psychology to participate as an advisory group to comment on our approach.

Process:

Members of the advisory group provided their feedback on the first version of the presentation format by completing a seven-item e-mail questionnaire. Two independent reviewers categorized the feedback according to five predefined criteria: “show stoppers” (ie, preventing further use), major problems (hindering further use, but where the participant eventually figured the problem out), minor problems/cosmetic issues, positive feedback, and suggestions for improvement. We focused particularly on showstoppers and major problems in the subsequent development process.

User Testing
Participants:

We recruited a convenience sample of physicians in six countries who spend a minimum of 50% of their time in patient care, with a diversity of academic backgrounds, settings (urban, rural), and years of experience.

Process:

Participants reviewed a clinical scenario and then used a tablet computer with access to a CPG excerpt relevant to the scenario, displayed in the new format. The participants engaged in a think-aloud protocol in which they were encouraged to state any and all thoughts as they reviewed the CPG. The interviewer supplemented this process with predefined questions exploring relevant facets of a modified version of Morville’s user experience model: findability, usefulness, usability, understandability, credibility, and desirability.32,33 We asked for feedback on the overall structure, layout, and components of the format. The audiotaped user test sessions lasted approximately 1 h.

The interviewers categorized participants’ responses according to the same criteria as applied for the advisory group feedback. After each round of user testing, we prepared summaries of the feedback and discussed suggested changes to implement in the next iteration. This iterative approach allows us to eliminate design flaws as soon as they were identified and to retest components apparently working well in later iterations with multiple participants.

Initial Format Development

Through brainstorming and sketching, we developed an electronic, multilayered presentation format for CPGs. The layered, stepwise approach allows health-care professionals to read recommendations first, with the option of accessing additional information on demand. This “top layer,” defined as the minimum information necessary for understanding and appropriately applying the recommendation in clinical encounters, consists of the following components:

  1. Recommendation statement including strength of the recommendation (strong vs weak).

  2. Rationale: a statement presenting the CPG panel’s reasoning regarding the key factors that drove the recommendation.

  3. Key information: the balance between benefits and harms of patient important outcomes, the confidence in the estimates of the effect of the interventions under consideration, extent of variability in patient values, and preferences and resource considerations.

Figure 2 shows the first top-layer version developed; Figures 3 and 4 display the revised versions after the first and second round of user testing, respectively.

Figure Jump LinkFigure 4 –  Current revised format. A, First layer with list of recommendations. B, Second layer with key information.Grahic Jump Location
Advisory Group Feedback

Of 30 invited stakeholders, 27 (90%) provided feedback on the first version. The main concern of respondents was the potentially overwhelming and confusing nature of methodologic concepts. Some stakeholders believed health-care professionals lack prior knowledge of the implications of weak and strong recommendations and would have difficulty in quickly grasping these concepts. There was no consensus on how to best convey the strength of the recommendation, with a minority of stakeholders suggesting to omit it altogether. Furthermore, there was no consensus on how to convey the estimates of effect, with suggestions ranging from changing the denominator from per 1,000 to per 100, displaying relative effects, percentages, and including CIs. Several stakeholders highlighted the need to focus on clinical relevance to achieve a balance between conveying methodologically correct and important issues while keeping the CPG information easily understood and useful. Suggestions included use of concise, actionable, and plain language and including practical information (eg, dosing for recommended treatments or contraindications).

User Testing With Physicians

We included 47 practicing physicians from six countries (Norway, Canada, United States, United Kingdom, Saudi Arabia, and Spain) in two rounds of user testing. Participants were either hospital-based (n = 24) or general practitioners (n = 23). Table 1 provides an overview of the main findings.

Table Graphic Jump Location
TABLE 1 ]  Participants’ Feedback to Specific Components of the Top Layer Format

Numeric data given as (No.; %) unless otherwise indicated.

a 

A think-aloud protocol was applied during the user test sessions, where participants were encouraged to express any sentiment. As a result, not all participants have provided explicit feedback to all individual components.

Morville’s Modified User Experience Facets: Combined Results From Rounds One and Two:
Usability and Findability:

The majority of participants perceived the structure and layout of the format to be relatively easy to use. They generally found the format to be somewhat crowded and suggested use of to-the-point advice, highlighting the patient population and applying decision algorithms when applicable. Participants perceived the color coding of the different strengths of the recommendations both helpful and visually appealing.

Usefulness and Desirability:

Participants considered the format to be useful overall. Several believed that growing familiarity over time would increase the format’s desirability and value. Several participants expressed that adding practical information (eg, dosage, contraindications or risk stratification scores) would increase usefulness. The provision of effect estimates was perceived as helpful and several stated they could use these estimates in a shared-decision making process with patients.

Credibility and Understandability:

Struggles with conceptual understanding of CPG methodology and perceiving the clinical usefulness of the evidence to recommendation framework dominated the feedback. During the first round of user testing, we found that participants either experienced a sense of uncertainty when faced with a weak recommendation (“I guess I wouldn’t even call this a recommendation”) and/or misinterpreted the strength of the recommendation to solely reflect the study design of the underlying evidence. As a consequence, we redesigned the format aiming to entice the participant to seek deeper explanatory layers of information, while still conveying the strength of the recommendation. During the second round, we tested different color-coded icons with an explanatory legend at the top. Most participants overlooked the legend and thus missed the fact that the icons reflected the strength of the recommendation.

We asked participants to suggest different ways of conveying the strength of the recommendation, but there were few suggestions and no clear consensus. A few suggested using letters and numbers (eg, 1A), currently used by some guideline organizations, however none could actually explain what the letters and numbers meant.

Two additional components resulted in uncertainty. In round one, we used the term “confidence in effect” when addressing the rating of the underlying evidence. This left several participants confused, and one participant interpreted it as being the guideline panel’s confidence in the recommendation. Simplifying the terminology in the second round, by changing the label to “quality of the evidence” and including reasons for rating down,34 resulted in increased acceptance and understanding of this key factor. The top-layer component, “values and preferences,” raised animosity during both rounds of user testing, participants finding the statements uninformative and lacking clinical relevance. Participants perceived this key factor as being a normative component, rather than referring to the values and preferences used by the panel to issue the recommendation.

We have developed and user tested a multilayered CPG format tailored to meet the information needs of health-care professionals, and our research represents an important first step in determining the optimal approach to presenting guidelines developed using the GRADE framework. The multilayered presentation format reflects key standards for trustworthy CPGs: ensuring transparency, adherence to rigorous methodology, meets research evidence supporting clarity, use of plain language, clinical applicability, and flexibility.2,3,14,3540 During the development process we have mainly focused on barriers to use, seeking to understand underlying reasons for these and potential paths to improvements, while at the same time holding on to our goal of communicating the evidence and rationale underlying the recommendations.

We uncovered serious issues during the first round of user testing. Extensive revisions made before the second round of user testing were partly successful, with the majority of participants expressing overall satisfaction with the revised format. Our experience in this process made one overarching challenge for effectively communicating CPGs to health-care professionals vividly evident: how to enable sufficient conceptual understanding of the underlying CPG methodology, while providing actionable recommendations and supporting evidence.

Conceptual Understanding of Methodology, Risk, and Uncertainty

The optimal way to convey key methodological concepts in trustworthy CPGs, such as recommendation strength, confidence in effect estimates (quality of evidence), and interpretation of effect estimates, remains uncertain.16,41 Color-coding appears to be one reasonable approach to conveying the strength of a recommendation. However, this will not be helpful for the color blind and for anyone using black and white printouts. Our attempt at using several icons failed, leaving clinicians confused. Therefore, we have yet to identify an approach superior to simply stating “strong recommendation” and “weak recommendation,” with a brief explanatory statement underneath (Fig 4).

Some evidence supports the usefulness of stating effect estimates as natural frequencies (eg, 10 in 100, or 100 in 1,000, rather than as a percentage or proportion), thereby always including a constant reference class and, furthermore, providing a brief narrative description of the magnitude of effect (gist or bottom line).15,17,19,4248 The results of our user testing support these strategies. We also observed an apparent enhanced understanding and comfort with confidence in effect estimates when changing the label to quality of the evidence and providing an explanation for the rating. However, we emphasize that we did not explicitly test participants’ conceptual understanding of the methodology behind the rating.34

We observed considerable discomfort with the key factor “values and preferences.” Our impression is that the discomfort stems as much from participants misunderstanding the implications of a weak recommendation, as from a direct dislike of the phrasing used. For example, participants found the statement shown in Figure 2 confusing, several stating they felt animosity toward having their patient’s preferences assumed and seemingly dictated by the guideline panel. Our formatting attempts were unsuccessful in communicating that values and preferences is not a normative component, but rather attempts to make the assumed values and preferences underlying the panels’ recommendations explicit. If clinicians gain a better understanding of what is being conveyed in weak and strong recommendations, their discomfort with “values and preference” might diminish.

Strengths and Limitations

Strengths of this study include the participation of a broad international group of stakeholders and end-user representatives. We used semi-interactive CPG mock-ups, allowing us to extensively test most facets of the user-experience model.

Our study has some limitations. Members of the research team performed some of the user tests, potentially biasing the collected feedback. The applicability of our results to other groups of health-care professionals other than physicians is uncertain and requires further study. Finally, we did not perform a participant validation of the summarized results.

Implications for Practice and Research

Our results are both encouraging and highlight the need for further research on how best to convey methodologic concepts. We are currently evaluating the multilayered format in surveys and randomized trials, investigating physicians’ understanding, attitudes, and preferences on CPGs through comparing traditional and new presentation formats. The format has been implemented in electronic guideline authoring and publication tools49,50 to enable real-life use, testing, and evaluation. A video demonstration can be accessed online. The novel format has already been adopted in a real guideline through the Norwegian adaptation of the American College of Chest Physicians (CHEST) antithrombotic guidelines,51 accessible to clinicians on any electronic device. Readers can view an English translation of two sample chapters from the guideline at www.magicapp.org/public.50

Video 1.

Multilayered Format

Author contributions: A. K. and P. O. V. had full access to all of the data in the study, and takes responsibility for the integrity of the data and the accuracy of the data analysis. A. K. was principal investigator on the study. A. K., L. B., P. A.-C., S. R., and P. O. V. contributed to the conception, design, and ethical approval of the study; A. K., L. B., P. A.-C., T. A., E. A. A., T. C., M. E., M. F., W. M., M. H. M., D. R., S. R., F. A. S., S. T., G. G., and P. O. V. contributed to writing the first draft of the article; and A. K., L. B., P. A.-C., T. A., E. A. A., T. C., M. E., M. F., W. M., M. H. M., D. R., S. R., F. A. S., S. T., G. G., and P. O. V. contributed to editing and approval of the final manuscript.

Role of sponsors: The sponsor had no role in the design of the study, the collection and analysis of the data, or the preparation of the manuscript.

Financial/nonfinancial disclosures: The authors have reported to CHEST the following conflicts of interest: Drs Kristiansen, Brandt, Guyatt, and Vandvik are members of the MAGIC research and innovation program. The remaining authors reported that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article.

Other contributions: We wish to extend a special thanks to Frankie Achille, interaction designer, for his extensive contributions in designing the presentation format. We further thank all members of the DECIDE Consortium Work Package 1 for contributing to the development of the presentation format. Finally, we would like to thank the National Institutes of Health Care and Excellence (NICE) for performing user testing.

Additional information: The Video can be found in the Multimedia section of the online article.

CPG

clinical practice guideline

DECIDE

Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence

GRADE

Grading of Recommendations Assessment, Development, and Evaluation

MAGIC

Making Grade the Irresistible Choice

Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261-1263. [CrossRef] [PubMed]
 
Michie S, Johnston M. Changing clinical behaviour by making guidelines specific. BMJ. 2004;328(7435):343-345. [CrossRef] [PubMed]
 
Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M, Lee M. Are nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision making. Health Serv Res. 2000;34(7):1429-1448. [PubMed]
 
Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217-224. [CrossRef] [PubMed]
 
Aboulsoud S, Huckson S, Wyer P, Lang E. Survey of preferred guideline attributes: what helps to make guidelines more useful for emergency health practitioners? Int J Emerg Med. 2012;5(1):42. [CrossRef] [PubMed]
 
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P; Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525-531. [CrossRef] [PubMed]
 
Graham R, Mancher M, Wolman DM. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011.
 
Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58. [PubMed]
 
Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172(21):1628-1633. [CrossRef] [PubMed]
 
Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301(8):868-869. [CrossRef] [PubMed]
 
Smith R. What clinical information do doctors need? BMJ. 1996:1062-1068.
 
Francke AL, Smit MC, de Veer AJE, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:38. [CrossRef] [PubMed]
 
Farquhar CM, Kofa EW, Slutsky JR. Clinicians’ attitudes to clinical practice guidelines: a systematic review. Med J Aust. 2002;177(9):502-506. [PubMed]
 
Shiffman RN, Dixon J, Brandt C, et al. BMC Medical Informatics and The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;25(3):1-8.
 
Akl EA, Oxman AD, Herrin J, et al. Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database Syst Rev. 2011;3(3):CD006776.
 
Akl EA, Maroun N, Guyatt G, et al. Symbols were superior to numbers for presenting strength of recommendations to health care consumers: a randomized trial. J Clin Epidemiol. 2007;60(12):1298-1305. [CrossRef] [PubMed]
 
Reyna V, Brainerd C. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learn Individ Differ. 2008;18(1):89-107. [CrossRef]
 
Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620-626. [CrossRef] [PubMed]
 
Spiegelhalter D, Pearson M, Short I. Visualizing uncertainty about the future. Science. 2011;333(6048):1393-1400. [CrossRef] [PubMed]
 
Langendam MW, Akl EA, Dahm P, Glasziou P, Guyatt G, Schünemann HJ. Assessing and presenting summaries of evidence in Cochrane Reviews. Syst Rev. 2013;2:81. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Kunz R, et al; GRADE Working Group. Going from evidence to recommendations. BMJ. 2008;336(7652):1049-1051. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [CrossRef] [PubMed]
 
Treweek S, Oxman AD, Alderson P, et al; DECIDE Consortium. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. [CrossRef] [PubMed]
 
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381-389. [CrossRef] [PubMed]
 
Creating trustworthy guidelines, evidence summaries and decision aids that we can all use and share. Magic Project website. http://www.magicproject.org. Accessed May 2014.
 
Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med. 2013;6(1):50-54. [CrossRef] [PubMed]
 
Vandvik PO, Lincoff AM, Gore JM, et al. Primary and secondary prevention of cardiovascular disease: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_ suppl):e637S-e668S. [PubMed]
 
You JJ, Singer DE, Howard PA, et al. Antithrombotic therapy for atrial fibrillation: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):e531S-e575S. [CrossRef] [PubMed]
 
The National Collaborating Centre for Chronic Conditions. Type 2 Diabetes National Clinical Guideline for Management in Primary and Secondary Care (Update). London, Englad: Royal College of Physicians; 2008. [PubMed] [PubMed]
 
Wroblewski L. Mobile First. La Vergne, TN: Ingram; 2011:123.
 
Blueprint. Groosoft website. http://www.groosoft.com/blueprint. Accessed January 2013.
 
Morville P. User experience design. 2004. Semantics Studio website. http://semanticstudios.com/publications/semantics/000029.php. Accessed December 2013.
 
Rosenbaum SE. Improving the User Experience of Evidence. A Design Approach to Evidence-Informed Health Care. Oslo, Norway: Oslo School of Architecture and Design; 2010:108-116.
 
Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [CrossRef] [PubMed]
 
Brouwers MC. Appraisal of Guidelines for Research & Evaluation II. AGREE website. http://www.agreetrust.org/resource-centre/agree-ii/. Accessed February 2014.
 
Gagliardi AR, Brouwers MC, Palda VA, Lemieux-Charles L, Grimshaw JM. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6(1):26. [CrossRef] [PubMed]
 
Michie S, Lester K. Words matter: increasing the implementation of clinical guidelines. Qual Saf Health Care. 2005;14(5):367-370. [CrossRef] [PubMed]
 
Lomotan EA, Michel G, Lin Z, Shiffman RN. How “should” we write guideline recommendations? Interpretation of deontic terminology in clinical practice guidelines: survey of the health services community. Qual Saf Health Care. 2010;19(6):509-513. [PubMed]
 
Codish S, Shiffman RN. A model of ambiguity and vagueness in clinical practice guideline recommendations. AMIA Annu Symp Proc. 2005;:146-150.
 
McGettigan P, Sly K, O’Connell D, Hill S, Henry D. The effects of information framing on the practices of physicians. J Gen Intern Med. 1999;14(10):633-642. [CrossRef] [PubMed]
 
Schünemann HJ, Best D, Vist G, Oxman AD; GRADE Working Group. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. CMAJ. 2003;169(7):677-680. [PubMed]
 
Estrada C, Barnes V, Collins C, Byrd JC. Health literacy and numeracy. JAMA. 1999;282(6):527. [CrossRef] [PubMed]
 
Perneger TV, Agoritsas T. Doctors and patients’ susceptibility to framing bias: a randomized trial. J Gen Intern Med. 2011;26(12):1411-1417. [CrossRef] [PubMed]
 
Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607-619. [CrossRef] [PubMed]
 
Woloshin S, Schwartz LM. Communicating data about the benefits and harms of treatment: a randomized trial. Ann Intern Med. 2011;155(2):87-96. [CrossRef] [PubMed]
 
Trevena LJ, Zikmund-Fisher BJ, Edwards A, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak. 2013;13(suppl 2):S7. [CrossRef] [PubMed]
 
Vandvik PO, Santesso N, Akl EA, et al. Formatting modifications in GRADE evidence profiles improved guideline panelists comprehension and accessibility to information. A randomized trial. J Clin Epidemiol. 2012;65(7):748-755. [CrossRef] [PubMed]
 
Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003;327(7417):741-744. [CrossRef] [PubMed]
 
GRADEPro. Guideline development tool. McMaster University and Evidence Prime Inc website. http://www.guidelinedevelopment.org/. Accessed May 2014.
 
MAGIC. Magic app website. http://www.magicapp.org/public. Accessed May 2014.
 
Kristiansen A, Brandt L, Agoritsas T, et al. Applying new strategies for the national adaptation, updating and dissemination of trustworthy guidelines: results from the Norwegian adaptation of the American College of Chest Physicians evidence-based guidelines on antithrombotic therapy and the prevention of thrombosis. 9th ed. Chest. 2014;146(3):735-761.
 

Figures

Figure Jump LinkFigure 1 –  Development method of the multilayered presentation format. RCT = randomized controlled trial.Grahic Jump Location
Figure Jump LinkFigure 2 –  Presentation format tested in round 1. A, First layer with list of recommendations. B, Second layer with key information. C, Rationale.Grahic Jump Location
Figure Jump LinkFigure 3 –  Presentation format tested in round 2. A, First layer with list of recommendations. B, Second layer with key information.Grahic Jump Location
Figure Jump LinkFigure 4 –  Current revised format. A, First layer with list of recommendations. B, Second layer with key information.Grahic Jump Location

Tables

Table Graphic Jump Location
TABLE 1 ]  Participants’ Feedback to Specific Components of the Top Layer Format

Numeric data given as (No.; %) unless otherwise indicated.

a 

A think-aloud protocol was applied during the user test sessions, where participants were encouraged to express any sentiment. As a result, not all participants have provided explicit feedback to all individual components.

Video 1.

Multilayered Format

References

Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261-1263. [CrossRef] [PubMed]
 
Michie S, Johnston M. Changing clinical behaviour by making guidelines specific. BMJ. 2004;328(7435):343-345. [CrossRef] [PubMed]
 
Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M, Lee M. Are nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision making. Health Serv Res. 2000;34(7):1429-1448. [PubMed]
 
Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians’ clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217-224. [CrossRef] [PubMed]
 
Aboulsoud S, Huckson S, Wyer P, Lang E. Survey of preferred guideline attributes: what helps to make guidelines more useful for emergency health practitioners? Int J Emerg Med. 2012;5(1):42. [CrossRef] [PubMed]
 
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P; Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525-531. [CrossRef] [PubMed]
 
Graham R, Mancher M, Wolman DM. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011.
 
Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58. [PubMed]
 
Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172(21):1628-1633. [CrossRef] [PubMed]
 
Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301(8):868-869. [CrossRef] [PubMed]
 
Smith R. What clinical information do doctors need? BMJ. 1996:1062-1068.
 
Francke AL, Smit MC, de Veer AJE, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:38. [CrossRef] [PubMed]
 
Farquhar CM, Kofa EW, Slutsky JR. Clinicians’ attitudes to clinical practice guidelines: a systematic review. Med J Aust. 2002;177(9):502-506. [PubMed]
 
Shiffman RN, Dixon J, Brandt C, et al. BMC Medical Informatics and The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;25(3):1-8.
 
Akl EA, Oxman AD, Herrin J, et al. Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database Syst Rev. 2011;3(3):CD006776.
 
Akl EA, Maroun N, Guyatt G, et al. Symbols were superior to numbers for presenting strength of recommendations to health care consumers: a randomized trial. J Clin Epidemiol. 2007;60(12):1298-1305. [CrossRef] [PubMed]
 
Reyna V, Brainerd C. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learn Individ Differ. 2008;18(1):89-107. [CrossRef]
 
Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620-626. [CrossRef] [PubMed]
 
Spiegelhalter D, Pearson M, Short I. Visualizing uncertainty about the future. Science. 2011;333(6048):1393-1400. [CrossRef] [PubMed]
 
Langendam MW, Akl EA, Dahm P, Glasziou P, Guyatt G, Schünemann HJ. Assessing and presenting summaries of evidence in Cochrane Reviews. Syst Rev. 2013;2:81. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Kunz R, et al; GRADE Working Group. Going from evidence to recommendations. BMJ. 2008;336(7652):1049-1051. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [CrossRef] [PubMed]
 
Treweek S, Oxman AD, Alderson P, et al; DECIDE Consortium. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. [CrossRef] [PubMed]
 
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381-389. [CrossRef] [PubMed]
 
Creating trustworthy guidelines, evidence summaries and decision aids that we can all use and share. Magic Project website. http://www.magicproject.org. Accessed May 2014.
 
Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med. 2013;6(1):50-54. [CrossRef] [PubMed]
 
Vandvik PO, Lincoff AM, Gore JM, et al. Primary and secondary prevention of cardiovascular disease: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_ suppl):e637S-e668S. [PubMed]
 
You JJ, Singer DE, Howard PA, et al. Antithrombotic therapy for atrial fibrillation: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):e531S-e575S. [CrossRef] [PubMed]
 
The National Collaborating Centre for Chronic Conditions. Type 2 Diabetes National Clinical Guideline for Management in Primary and Secondary Care (Update). London, Englad: Royal College of Physicians; 2008. [PubMed] [PubMed]
 
Wroblewski L. Mobile First. La Vergne, TN: Ingram; 2011:123.
 
Blueprint. Groosoft website. http://www.groosoft.com/blueprint. Accessed January 2013.
 
Morville P. User experience design. 2004. Semantics Studio website. http://semanticstudios.com/publications/semantics/000029.php. Accessed December 2013.
 
Rosenbaum SE. Improving the User Experience of Evidence. A Design Approach to Evidence-Informed Health Care. Oslo, Norway: Oslo School of Architecture and Design; 2010:108-116.
 
Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [CrossRef] [PubMed]
 
Brouwers MC. Appraisal of Guidelines for Research & Evaluation II. AGREE website. http://www.agreetrust.org/resource-centre/agree-ii/. Accessed February 2014.
 
Gagliardi AR, Brouwers MC, Palda VA, Lemieux-Charles L, Grimshaw JM. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6(1):26. [CrossRef] [PubMed]
 
Michie S, Lester K. Words matter: increasing the implementation of clinical guidelines. Qual Saf Health Care. 2005;14(5):367-370. [CrossRef] [PubMed]
 
Lomotan EA, Michel G, Lin Z, Shiffman RN. How “should” we write guideline recommendations? Interpretation of deontic terminology in clinical practice guidelines: survey of the health services community. Qual Saf Health Care. 2010;19(6):509-513. [PubMed]
 
Codish S, Shiffman RN. A model of ambiguity and vagueness in clinical practice guideline recommendations. AMIA Annu Symp Proc. 2005;:146-150.
 
McGettigan P, Sly K, O’Connell D, Hill S, Henry D. The effects of information framing on the practices of physicians. J Gen Intern Med. 1999;14(10):633-642. [CrossRef] [PubMed]
 
Schünemann HJ, Best D, Vist G, Oxman AD; GRADE Working Group. Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations. CMAJ. 2003;169(7):677-680. [PubMed]
 
Estrada C, Barnes V, Collins C, Byrd JC. Health literacy and numeracy. JAMA. 1999;282(6):527. [CrossRef] [PubMed]
 
Perneger TV, Agoritsas T. Doctors and patients’ susceptibility to framing bias: a randomized trial. J Gen Intern Med. 2011;26(12):1411-1417. [CrossRef] [PubMed]
 
Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607-619. [CrossRef] [PubMed]
 
Woloshin S, Schwartz LM. Communicating data about the benefits and harms of treatment: a randomized trial. Ann Intern Med. 2011;155(2):87-96. [CrossRef] [PubMed]
 
Trevena LJ, Zikmund-Fisher BJ, Edwards A, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak. 2013;13(suppl 2):S7. [CrossRef] [PubMed]
 
Vandvik PO, Santesso N, Akl EA, et al. Formatting modifications in GRADE evidence profiles improved guideline panelists comprehension and accessibility to information. A randomized trial. J Clin Epidemiol. 2012;65(7):748-755. [CrossRef] [PubMed]
 
Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003;327(7417):741-744. [CrossRef] [PubMed]
 
GRADEPro. Guideline development tool. McMaster University and Evidence Prime Inc website. http://www.guidelinedevelopment.org/. Accessed May 2014.
 
MAGIC. Magic app website. http://www.magicapp.org/public. Accessed May 2014.
 
Kristiansen A, Brandt L, Agoritsas T, et al. Applying new strategies for the national adaptation, updating and dissemination of trustworthy guidelines: results from the Norwegian adaptation of the American College of Chest Physicians evidence-based guidelines on antithrombotic therapy and the prevention of thrombosis. 9th ed. Chest. 2014;146(3):735-761.
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Find Similar Articles
CHEST Journal Articles
  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543