0
Original Research: Education, Research, And Quality Improvement |

Adaptation of Trustworthy Guidelines Developed Using the GRADE MethodologyFive-Step Adaptation Process for GRADE Guidelines: A Novel Five-Step Process FREE TO VIEW

Annette Kristiansen, MD; Linn Brandt, MD; Thomas Agoritsas, MD; Elie A. Akl, MD, PhD, MPH; Eivind Berge, MD, PhD; Johan Bondi, MD, PhD; Anders E. Dahm, MD, PhD; Lars-Petter Granan, MD, PhD; Sigrun Halvorsen, MD, PhD; Pål-Andre Holme, MD, PhD; Anne Flem Jacobsen, MD, PhD; Eva-Marie Jacobsen, MD, PhD; Ignacio Neumann, MD; Per Morten Sandset, MD, PhD; Torunn Sætre, MD, PhD; Arnljot Tveit, MD, PhD; Trond Vartdal, MD, PhD; Gordon Guyatt, MD, FCCP; Per Olav Vandvik, MD, PhD
Author and Funding Information

From the Department of Internal Medicine (Drs Kristiansen, Brandt, and Vandvik), Innlandet Hospital Trust, Gjøvik, Norway; Institute for Health and Society (Drs Kristiansen, Brandt, and Vandvik), Faculty of Medicine, and Institute of Clinical Medicine (Dr Sandset), University of Oslo, Oslo, Norway; Department of Internal Medicine (Drs Kristiansen and Brandt), Diakonhjemmet Hospital, Oslo, Norway; Department of Clinical Epidemiology and Biostatistics (Drs Agoritsas, Akl, Neumann, and Guyatt), McMaster University, Hamilton, ON, Canada; Department of Internal Medicine (Dr Akl), American University of Beirut, Beirut, Lebanon; Department of Internal Medicine (Dr Berge), Oslo University Hospital, Ullevål, Oslo, Norway; Department of Surgery (Dr Bondi), Department of Medical Research (Dr Tveit), and Department of Internal Medicine (Dr Vartdal), Bærum Hospital, Vestre Viken Hospital, Drammen, Norway; Department of Hematology (Drs Dahm, Holme, Jacobsen, Sandset), Department of Physical Medicine and Rehabilitation (Dr Granan), Department of Pain Management and Research (Dr Granan), Division of Emergencies and Critical Care, Department of Cardiology B (Dr Halvorsen), Department of Obstetrics and Gynaecology (Dr Flem Jacobsen), and Center for Vascular Surgery (Dr Sætre), Oslo University Hospital, Oslo, Norway; Oslo Sports Trauma Research Center (Dr Granan), Norwegian School of Sport Sciences, Oslo, Norway; and Norwegian Knowledge Centre for the Health Services (Dr Vandvik), Oslo, Norway.

CORRESPONDENCE TO: Annette Kristiansen, MD, Department of Internal Medicine, Sykehuset Innlandet HF, Gjøvik, Norway; e-mail: annettekrist@gmail.com


Part of this article has been presented in poster form at the 2012 G-I-N Conference, August 22-25, 2012, Berlin, Germany.

FUNDING/SUPPORT: Innlandet Hospital Trust, the Southern and Eastern Norway Regional Health Authority, and the Norwegian Research Council have provided research grants. The Norwegian Medical Association has provided grants to support completion of the adaptation process.

Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details.


Chest. 2014;146(3):727-734. doi:10.1378/chest.13-2828
Text Size: A A A
Published online

BACKGROUND:  Adaptation of guidelines for use at the national or local level can facilitate their implementation. We developed and evaluated an adaptation process in adherence with standards for trustworthy guidelines and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology, aiming for efficiency and transparency. This article is the first in a series describing our adaptation of Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines for a Norwegian setting.

METHODS:  Informed by the ADAPTE framework, we developed a five-step adaptation process customized to guidelines developed using GRADE: (1) planning, (2) initial assessment of the recommendations, (3) modification, (4) publication, and (5) evaluation. We developed a taxonomy for describing how and why recommendations from the parent guideline were modified and applied a mixed-methods case study design for evaluation of the process.

RESULTS:  We published the adapted guideline in November 2013 in a novel multilayered format. The taxonomy for adaptation facilitated transparency of the modification process for both the guideline developers and the end users. We excluded 30 and modified 131 of the 333 original recommendations according to the taxonomy and developed eight new recommendations. Unforeseen obstacles related to acquiring a licensing agreement and procuring a publisher resulted in a 9-month delay. We propose modifications of the adaptation process to overcome these obstacles in the future.

CONCLUSIONS:  This case study demonstrates the feasibility of a novel guideline adaptation process. Replication is needed to further validate the usefulness of the process in increasing the organizational and methodologic efficiency of guideline adaptation.

Figures in this Article

Developing trustworthy evidence-based clinical practice guidelines (CPGs) requires considerable expertise, time, and financial support. For instance, the development of the Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (AT9)1 involved 128 methodologists and clinical experts, in addition to dedicated staff, to adduce 15 management chapters over a 2-year period. AT9 was produced by rigorous application of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework, the leading methodology for evaluating evidence and developing CPGs.24

Developing trustworthy guidelines is challenging, and implementation of recommendations at the point of care presents additional challenges,5 one of these being that guidelines developed for one practice setting may have serious limitations in their application to another setting. Thus, adaptation of international CPGs for use at the national and local level can improve their implementation.6,7 Adaptation reduces language barriers and takes specific national contexts into account while limiting unnecessary duplication.

Recognizing the need for antithrombotic guidelines in Norway and being aware of the forthcoming publication of the AT9 guidelines, a group of thrombosis and guideline methodology experts decided to adapt this international guideline. The task, then, was to develop an optimal methodology for the adaptation process.

ADAPTE is a widely used framework for adaptation developed by the ADAPTE collaboration and later transferred to the Guideline International Network.6,8 By choosing AT9, we could bypass the initial steps proposed by ADAPTE to identify optimal CPGs for adaptation and instead develop and evaluate an adaptation process and taxonomy for modifying recommendations customized to GRADE guidelines. A particular goal of the process was the implementation of a new, multilayered presentation format using a recently developed electronic guideline platform.9,10

This article is the first in a series of three featured in CHEST. Here, we present details about the adaptation process. The second article11 provides an account of the end results of the adaptation of AT9 to a Norwegian setting, and the third12 describes the use of the McMaster PLUS (Premium Literature Search) database for providing evidence to inform updates.

The adaptation process detailed herein is based on the ADAPTE and GRADE frameworks3,4,6,13 as well as on our personal experience from previous CPG development. We used the GRADE methodology as a template when outlining the taxonomy for modifications and applied a mixed-methods approach with both qualitative and quantitative components to refine and evaluate the adaptation process.

Choice of the Parent Guideline for Adaptation to Norway

We selected AT9 because it is an authoritative international CPG that, at the time we began, was current (published in February 2012). Furthermore, AT9 is the largest CPG to rigorously apply the GRADE methodology, providing authoritative assessments of confidence in evidence and explicit rationales for the strength of its recommendations.1 Finally, AT9 informs practice in a wide variety of clinical contexts (eg, hematology, surgery, cardiology, obstetrics).14

Selection of Panelists

We asked relevant medical specialty societies to nominate candidate panelists. We recruited 42 content experts, 11 of whom were designated as chapter editors and took primary responsibility for completing the adapted chapters. Each panelist reported time spent on the adaptation work.

Evaluation

Chapter editors provided a formal evaluation at the end of the adaptation. A person not otherwise affiliated with the adaptation work performed a short semistructured interview of four out of 11 chapter editors by teleconference, audiotaping and transcribing the interviews. The editors were asked to score their overall experience on a scale from 1 to 10, provide suggestions for improvements of the framework, and state positive and negative experiences. The interviewer and one member of the editorial committee (see step 1 in the Results section) reviewed the transcribed responses. We distributed the summaries to all 11 chapter editors for verification.

We performed the adaptation work between October 2011 and November 2013. In the following sections, we describe the five-step adaptation process and related taxonomy that we initially proposed as well as the modifications made as the adaptation progressed.

Timeline

We outlined an ambitious timeline aiming at final publication by the fall of 2012, 1 year after starting the adaptation, but we faced major obstacles to reaching that goal (Fig 1). We initially planned to develop the adapted AT9 as a national guideline under a mandate from the Norwegian Directorate of Health. In October 2012, the editorial committee became aware of the need for a licensing agreement with the American College of Chest Physicians (CHEST) for permission to use AT9 as the template for the Norwegian adaptation. The directorate was prohibited from entering into such an agreement with an American nongovernmental agency, and thus, the collaboration came to an end. This development spurred one of the guideline group leaders (P. M. S.) to found the Norwegian Society of Thrombosis and Haemostasis under the umbrella of the Norwegian Medical Association. The society ultimately signed a licensing agreement with CHEST and is the formal publisher of the adapted guideline. Reaching an agreement delayed the adaptation process by 9 months.

Figure Jump LinkFigure 1  Timeline of the Norwegian adaptation of AT9. AT9 = Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines; GRADE = Grading of Recommendations Assessment, Development and Evaluation.Grahic Jump Location
Five-Step Adaptation Process
Step 1: Planning:
Establish an Editorial Committee:

One individual with expertise in GRADE and guideline development (P. O. V.) and a content expert on thrombosis (P. M. S.) led the editorial committee, which comprised three method experts and four content experts whose responsibilities were to oversee and ensure timely completion of the adaptation work. The editorial committee informed the AT9 leadership and chapter editors of the adaptation endeavor.

Choose Topics to Adapt:

The editorial committee chose 11 of 15 management chapters in AT9 for adaptation, excluding three chapters due to resource constraints. The chapter addressing antithrombotic and thrombolytic therapy for valvular disease will be adapted in 2014.

Choose Chapter Editors:

The 11 chapter editors are frontline clinicians with a PhD degree and content expertise in the relevant subject area. None had prior experience with CPG development; one had experience with GRADE for systematic reviews. All editors underwent formal training in practical application of the GRADE methodology prior to commencing the adaptation work.

Convene Start-up Conference:

We organized a daylong conference providing an introduction to evidence-based practice, trustworthy CPG development with GRADE, and an explanation of the adaptation process.

Plan Implementation Strategies:

We undertook a multifaceted approach to boost dissemination and implementation. To ensure ownership and acceptance, a diverse set of stakeholders participated in the guideline work both as panelists and as peer reviewers. We rewrote and restructured the original content in a multilayered format, making it available on various electronic platforms. Presentations at annual meetings of each specialty society and articles in the two leading medical journals in Norway raised the profile of the guideline. We reviewed other national and European guidelines1521 currently in use to identify discrepancies with AT9.

Step 2: Initial Assessment of the Recommendations:

The panelists reviewed each recommendation in their designated chapter and formally recorded their views regarding whether the recommendation could stand as it was or whether there was a need for modification, exclusion, or development of new recommendations. A structured spreadsheet, organized according to the taxonomy (Fig 2), facilitated this process. The panels used these written records as a basis for discussions, deliberations, and final decisions.

Figure Jump LinkFigure 2  Structured application of the taxonomy. The panel member first decides whether to keep or exclude the recommendation. If the recommendation is retained, the need for modification is assessed. If the recommendation is modified, the panel member recognizes and explicitly states which, if any, of the five factors about the underlying evidence he or she disagrees with, including the clinical question (PICO-question) documentation, assessment of the evidence, applied baseline risk, and relative risk. The panel member then considers the deliberations made when moving from evidence to recommendation (disagreement with the balancing of the benefits and harms of the interventions and, thus, the assumed typical patient preferences and values; the overall grading of the quality of the evidence or national or local resource issues either societal or patient related). After discussions within the panel, the final mode of modification is stated before writing the modified recommendation. PICO = patient, intervention, comparator, outcome; rec = recommendation.Grahic Jump Location
Conflicts of Interest:

The panelists reported financial and intellectual conflicts of interest (COIs)22,23 for any anticipated modified, excluded, or new recommendation but not for recommendations that were unchanged. A table depicting the panelists’ COIs is provided in the background section of each chapter.

Step 3: Modification:
Updated Search for New Documentation:

For a literature update, we used the McMaster PLUS database, which continuously collects high-quality, preappraised studies.24 The chapter editors reviewed the search results, assessed the new documentation, and, together with a methods expert, determined the relevance of each article.

The various panels translated, restructured, and rewrote the recommendations in a new presentation format. Each systematically reevaluated four factors3 that determined the direction and strength of the recommendations: the balance between benefits and harms, confidence in the estimates of the effect of the interventions under consideration, extent of assumed variability in patient values and preferences, and resource considerations. If the panel decided to exclude or modify a recommendation, a more extensive reassessment of the underlying evidence, including the tabular evidence summaries provided in AT9, was mandatory. The panels consulted with the editors of AT9 on content issues as needed.

The panels modified about one-third of the 333 recommendations under consideration from AT9 (Fig 3). Often, the modifications were motivated by multiple factors in the taxonomy. During the course of the adaptation, we added feasibility to the taxonomy of reasons for changing recommendations because many modifications were due to lacking availability or tradition for use of interventions in Norway. In these circumstances, the panels consistently recommended readily available treatment options with similar risk-benefit profiles. We also added major changes to the wording because contentwise, many of the alterations were minor and consisted of modifying the recommendation phrasing to make the recommendations more specific and actionable.

Figure Jump LinkFigure 3  Overview of modifications according to the taxonomy. 119 due to lacking applicability; two due to low prevalence; and nine deemed redundant as isolated recommendations, but often the information was included in the general introduction or under practical information. 2Most new recommendations were developed due to new and updated documentation (see the second article in this series11 for more details). 3An evidence profile is a table providing an overview of the PICO question, including only patient-important outcomes, the relative and absolute estimates of effect, and a rating of the quality of evidence according to five factors (risk of bias, heterogeneity, imprecision, indirectness, and publication bias). The 35 new or modified evidence profiles provided new absolute effect estimates due to altered baseline risks or relative effects. These new absolute estimates resulted in six recommendations changing direction and four changing in strength and in the addition of seven new recommendations. 4Often, one or more interventions or comparators were excluded due to feasibility issues (eg, medications not being readily available or infrequently used in Norway and equal options being suggested instead). 5Five recommendations were modified due to new baseline documentation from a Norwegian or Scandinavian population. Twenty-six recommendations were modified partly due to updated documentation published after November 2010. For the chapters on major orthopedic surgery and pregnancy, the relative effect of low-molecular-weight heparin vs placebo as thromboprophylaxis was derived from a meta-analysis by Collins et al cited in the AT9 chapter on the prevention of VTE in nonorthopedic surgical patients.25 6Baseline risk refers to the risk in untreated patients. 7Disagreement with how the balance between the benefits and harms of an intervention is valued resulting from disagreement with the panel’s assumption of typical patient preferences and values. 8Disagreement with the assessment of the overall quality of the evidence across individual outcomes. 9Disagreement with the assessment of resource use, cost-effectiveness, and so forth. 10Eight recommendations were modified from strong to weak, whereas the remaining 11 were modified from weak to strong. 11A PICO element is excluded, added, or modified, for example, exclusion of intermittent pneumatic compression device as one of several thromboprophylactic options, adding apixaban to dabigatran as a treatment option in patients with atrial fibrillation, and modifying where AT9 states vitamin K antagonists to warfarin in the adapted guideline because this is the only available vitamin K antagonist in Norway. See Figure 1 and 2 legends for expansion of abbreviations.Grahic Jump Location
Submission of Final Draft to the Editorial Committee:

One content expert and one methods expert on the editorial committee reviewed each chapter.

Peer Review:

We invited all relevant medical specialty organizations, the Norwegian Directorate of Health, the Norwegian Medicines Agency, and representatives of CHEST to provide comments and feedback. Seven of the 15 invited organizations provided written responses. Both the individual panels and the editorial committee reviewed and discussed the responses and reached consensus on any modifications made, most of which were minor.

Step 4: Publication:

We published the adapted guideline online in November 2013 through a newly developed web authoring and publication platform,10,26 including offline access on smartphones and tablets.

Step 5: Evaluation and Planning for the Future:
Evaluation:

The chapter editors gave an average score of 7 of 10 for the overall adaptation process. Positive experiences included learning the GRADE methodology and improving and tailoring the recommendations on the basis of current best evidence and national circumstances. The chapter editors expressed frustration by the long delay in the timeline resulting from insufficient planning and recommended that this step be improved in future endeavors.

Future Updates:

We plan to dynamically update the recommendations at least every 3 months by the same team that produced the adapted guideline. During the first year, the updates will be performed within the scope of a separate research project27 comparing exhaustive and restricted literature monitoring strategies and assessing whether dynamically updating a guideline is feasible and which strategy is both sufficiently comprehensive and time efficient. The editorial committee is, however, currently in negotiations with CHEST regarding a separate licensing agreement detailing future updates, implying that the current plan might be subject to alterations. The online presentation of the recommendations provides end users with the opportunity to comment on each individual recommendation.

Requirements
Time:

The panelists’ reports of time spent on the guideline were variable. The 11 chapter editors spent a median of 116 h (range, 50-218 h). Of the remaining 31 panelists, only 18 reported time spent, with a median of 12.5 h (range, 3-60 h).

Human Resources:

The panelists are full-time clinicians, some adjunct or associate professors, working on the adaptation without remuneration. Separate research grants funded the three methods experts, including the senior methodologist and head of the guideline adaptation initiative (P. O. V.). As part of the editorial committee, the methods experts spent a substantial amount of time not only assisting the chapter editors on understanding and evaluating the content but also editing the final chapters.

Additional Expenses:

We spent approximately $20,000 on conferences and meetings and travel expenses. This figure excludes publication costs.

Through this project, we have had a unique opportunity to provide patients in Norway with evidence-based care according to authoritative international guidelines tailored to national circumstances. We developed and tested a new adaptation process and taxonomy that facilitates a structured adherence to updated standards for CPGs.22,28 The road to final publication proved somewhat cumbersome not because of problems at the level of review of the recommendations but, rather, because of administrative issues related to permission to publish.

The panels modified approximately one-third of the original AT9 recommendations, often by rewriting the recommendations to make them more specific and actionable. We consulted other guidelines currently in extensive use to check for any practice-altering recommendations. The few recommendations that we believe could be somewhat controversial have been provided with extensive rationales. All these efforts were performed to increase the uptake and feasibility of the recommendations.29

The close, hands-on collaboration between the content and methods experts proved to be an appreciable advantage, allowing for the rapid resolution of complex problems and, therefore, avoiding unnecessary time loss. We recommend that participants with methodological expertise be included on the editorial committee and in guideline development whenever possible.

Given current standards of most guidelines,30,31 identifying trustworthy CPGs to adapt might prove to be somewhat difficult. There is, however, a growing international awareness of the need for quality standards for guidelines,22 with both the Institute of Medicine and Guidelines International Network heavily invested in increasing the quality of guidelines.

Previous adaptation endeavors have proven to be time and resource consuming, and suggestions have been put forth to centralize resources and place a larger emphasis on disseminating strategies.7,32,33 The availability of a current, high-quality, trustworthy CPG as the basis for our adaptation proved crucial to efficient adaptation. Our pragmatic approach of not requiring a complete analysis of the underlying evidence for recommendations not modified further increases the feasibility and efficiency of the process. Embedding the framework and taxonomy in an electronic authoring and publication platform can potentially save time, especially if the parent guideline has been developed using that very same platform. Conscientious use of the pathway for deciding on keeping or altering recommendations (Fig 2) will ensure a systematic and transparent process that is reproducible and true to standards for trustworthy guidelines and the GRADE methodology, making the recommendations more easily adaptable for other guideline groups with limited resources.

Limitations

We opted for a pragmatic approach when handling COIs and only reported these for modified, excluded, and new recommendations. We recognize that this can potentially introduce bias into the adaptation process and suggest that others follow updated standards by collecting COIs at the start of the project.

We invited 28 guideline developers from Europe and North America who are familiar with the GRADE methodology to provide feedback on the proposed process and taxonomy. However, only two replied, and thus, we have not been able to make modifications to the process based on stakeholder feedback.

Conclusions and Future Research

Our experience was overall successful in nationally adapting a large and challenging CPG that included 333 recommendations. Replication of the process and use of guidelines for decision-making regarding adaptation (the adaptation taxonomy) are needed to further increase efficiency in guideline adaptation and appropriate use of resources.

Other guideline groups may benefit from replicating our methodology. We have included the five-step adaptation process in our Guideline Authoring and Publication Platform as a checklist and plan to incorporate the taxonomy as a separate feature. We are happy to provide guidance and collaborate on the assessment of future adaptations. Please refer to www.magicproject.org for a more extensive outline of the adaptation process and contact information.

Author contributions: A. K. and P. O. V. had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. A. K. was the principal investigator. A. K. contributed to the study concept, design, and ethical approval, drafting of the manuscript, and editing and final approval of the manuscript; L. B., T. A., E. A. A., E. B., J. B., A. E. D., L.-P. G., S. H., P.-A. H., A. F. J., E.-M. J., I. N., P. M. S., T. S., A. T., T. V., and G. G. contributed to the editing and final approval of the manuscript; and P. O. V. contributed to the study concept, design, and ethical approval and editing and final approval of the manuscript.

Financial/nonfinancial disclosures: The authors have reported to CHEST the following conflicts of interest: Dr Dahm has received speaking honoraria from Pfizer, Inc. Drs Akl, Guyatt, and Vandvik participated in the writing of the American College of Chest Physicians original guideline (Antithrombotic Therapy and Prevention of Thrombosis, 9th ed). Dr Halvorsen has received speakers honoraria from AstraZeneca plc, Eli Lilly and Co, Bayer AG, Boehringer Ingelheim GmbH, and Bristol-Myers Squibb. Dr Tveit has acted as a consultant for Bayer AG, Boehringer Ingelheim GmbH, and Pfizer Inc; lectured for Boehringer Ingelheim GmbH, Nycomed, and Pfizer Inc; and has been a member of advisory committees for Bayer AG, Boehringer Ingelheim GmbH, Bristol-Myers Squibb, Merck Sharp & Dohme Corp, and sanofi-aventis US, LLC. Drs Kristiansen, Brandt, Vandvik, and Guyatt are members of the nonprofit organization MAGIC. Drs Agoritsas, Berge, Bondi, Granan, Halvorsen, Holme, Flem Jacobsen, Jacobsen, Neumann, Sandset, Sætre, and Vartdal have reported that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article.

Role of sponsors: The sponsors had no role in the design of the study, the collection and analysis of the data, or the preparation of the manuscript.

Other contributions: The authors thank Pablo Alonso-Coello, MD, PhD, for contributing to the development of the adaptation process. The authors also thank all the panel members for helping to complete the Norwegian adaptation: Ola Røkke, MD, PhD; Arild Horn, MD, PhD; Jon Olav Drogset; MD, PhD; Olav Arnulv Asserson, MD; Lars Engebretsen MD, PhD; John Bjarne Hansen, MD; Ellen Elisabeth Brodin, MD, PhD; Anne Mette Njaastad, MD; Petter Quist Paulsen, MD, PhD; Waleed Ghanima, MD, PhD; Sara Ulimoen, MD; Jan Pål Loennechen, MD; Jørgen Smith, MD; Hanne Ellekjær, MD, PhD; Halvor Næss, MD, PhD; Ellisiv Mathiesen, MD, PhD; Bjørn Bendz, MD, PhD; Jan Eritsland, MD, PhD; Beate Viddal, MD; Aslak Holtestaul, MD; Martin Altreuther, MD; Ole Jørgen Grøtta, MD; Henrik Steenfeldt Foss, MD; Jørgen Jørgensen, MD, PhD; Line Bjørge, MD, PhD; Anette Løken Eilertsen, MD, PhD; Tore Henriksen, MD, PhD; Harald Breivik, MD; Tom Nilsen Hoel, MD, PhD; Arne Fetveit, MD; and Inger Anne Næss, MD, PhD.

AT9

Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines

CHEST

American College of Chest Physicians

COI

conflict of interest

CPG

clinical practice guideline

GRADE

Grading of Recommendations Assessment, Development, and Evaluation

Guyatt GH, Norris SL, Schulman S, et al. Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):53S–-70S.
 
Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Kunz R, et al; GRADE Working Group. Going from evidence to recommendations. BMJ. 2008;336(7652):1049-1051. [CrossRef] [PubMed]
 
Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med. 2013;6(1):50-54. [CrossRef] [PubMed]
 
Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):1-72.
 
Fervers B, Burgers JS, Haugh MC, et al. Adaptation of clinical guidelines: literature review and proposition for a framework and procedure. Int J Qual Health Care. 2006;18(3):167-176. [CrossRef] [PubMed]
 
Harrison MB, Graham ID, van den Hoek J, Dogherty EJ, Carley ME, Angus V. Guideline adaptation and implementation planning: a prospective observational study. Implement Sci. 2013;8:49. [CrossRef] [PubMed]
 
ADAPTE Collaboration. The ADAPTE Process: Resource Toolkit for Guideline Adaptation. Version 2.0. Guideline International Network website. www.g-i-n.net. Accessed November 2013.
 
Treweek S, Oxman AD, Alderson P, et al; DECIDE Consortium. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. [CrossRef] [PubMed]
 
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381-389. [CrossRef] [PubMed]
 
Kristiansen A, Brandt L, Agoritsas T, et al. Applying new strategies for the national adaptation, updating, and dissemination of trustworthy guidelines: results from the Norwegian adaptation of the antithrombotic therapy and the prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2014;146(3):735-761.
 
Agoritsas T, Kristiansen A, Agarwal A, et al. Updating and monitoring of clinical practice guidelines: performance of McMaster Premium LiteratUre Service (PLUS) in identifying new key studies. Chest. In press. doi:10.1378/chest.14-1366.
 
Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [CrossRef] [PubMed]
 
Guyatt GH, Akl EA, Crowther M, Gutterman DD, Schuünemann HJ; for the American College of Chest Physicians Antithrombotic Therapy and Prevention of Thrombosis Panel. Executive summary: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):7S–-47S.
 
Dick F, Ricco JB, Davies AH, et al. Chapter VI: follow-up after revascularisation. Eur J Vasc Endovasc Surg. 2011;42(suppl 2):S75-S90. [CrossRef] [PubMed]
 
Hamm CW, Bassand J-P, Agewall S, et al; ESC Committee for Practice Guidelines. ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: the Task Force for the Management of Acute Coronary Syndromes (ACS) in Patients Presenting Without Persistent ST-Segment Elevation of the European Society of Cardiology (ESC). Eur Heart J. 2011;32(23):2999-3054. [CrossRef] [PubMed]
 
Gaglia MA Jr, Waksman R. Systematic review of thienopyridine discontinuation and its impact upon clinical outcomes. Eur Heart J. 2011;32(19):2358-2364. [CrossRef] [PubMed]
 
The Norwegian Directorate of Health. Nasjonale Retningslinjer for individuell primærforebygging av hjerte- og karsykdommer. Oslo, Norway: The Norwegian Directorate of Health; 2009.
 
The Norwegian Directorate of Health. Behandling og rehabilitering ved hjerneslag. Oslo, Norway: The Norwegian Directorate of Health; 2010.
 
Debourdeau P, Farge D, Beckers M, et al. International clinical practice guidelines for the treatment and prophylaxis of thrombosis associated with central venous catheters in patients with cancer. J Thromb Haemost. 2013;11(1):71-80. [CrossRef] [PubMed]
 
Farge D, Debourdeau P, Beckers M, et al. International clinical practice guidelines for the treatment and prophylaxis of venous thromboembolism in patients with cancer. J Thromb Haemost. 2013;11(1):56-70. [CrossRef] [PubMed]
 
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P; Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525-531. [CrossRef] [PubMed]
 
Lenzer J. Why we can’t trust clinical guidelines. BMJ. 2013;346:f3830. [CrossRef] [PubMed]
 
Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews. J Clin Epidemiol. 2012;65(1):62-72. [CrossRef] [PubMed]
 
Gould MK, Garcia DA, Wren SM, et al. Prevention of VTE in nonorthopedic surgical patients: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):e227S-e277S. [PubMed]
 
MAGIC. Public guidelines. MAGIC app website. http://www.magicapp.org/public. Accessed December 2013.
 
MAGIC. DYNAM-IT. MAGIC website. http://www.magicproject.org/dynam-it. Accessed December 2013.
 
Graham R, Mancher M, Wolman DM. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011.
 
Shiffman RN, Dixon J, Brandt C, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;5:23. [CrossRef] [PubMed]
 
Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58. [PubMed]
 
Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172(21):1628-1633. [CrossRef] [PubMed]
 
Silagy CA, Weller DP, Lapsley H, Middleton P, Shelby-James T, Fazekas B. The effectiveness of local adaptation of nationally produced clinical practice guidelines. Fam Pract. 2002;19(3):223-230. [CrossRef] [PubMed]
 
Burgers JS, Anzueto A, Black PN, et al; ATS/ERS Ad Hoc Committee on Integrating and Coordinating Efforts in COPD Guideline Development. Adaptation, evaluation, and updating of guidelines: article 14 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):304-310. [CrossRef] [PubMed]
 

Figures

Figure Jump LinkFigure 1  Timeline of the Norwegian adaptation of AT9. AT9 = Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines; GRADE = Grading of Recommendations Assessment, Development and Evaluation.Grahic Jump Location
Figure Jump LinkFigure 2  Structured application of the taxonomy. The panel member first decides whether to keep or exclude the recommendation. If the recommendation is retained, the need for modification is assessed. If the recommendation is modified, the panel member recognizes and explicitly states which, if any, of the five factors about the underlying evidence he or she disagrees with, including the clinical question (PICO-question) documentation, assessment of the evidence, applied baseline risk, and relative risk. The panel member then considers the deliberations made when moving from evidence to recommendation (disagreement with the balancing of the benefits and harms of the interventions and, thus, the assumed typical patient preferences and values; the overall grading of the quality of the evidence or national or local resource issues either societal or patient related). After discussions within the panel, the final mode of modification is stated before writing the modified recommendation. PICO = patient, intervention, comparator, outcome; rec = recommendation.Grahic Jump Location
Figure Jump LinkFigure 3  Overview of modifications according to the taxonomy. 119 due to lacking applicability; two due to low prevalence; and nine deemed redundant as isolated recommendations, but often the information was included in the general introduction or under practical information. 2Most new recommendations were developed due to new and updated documentation (see the second article in this series11 for more details). 3An evidence profile is a table providing an overview of the PICO question, including only patient-important outcomes, the relative and absolute estimates of effect, and a rating of the quality of evidence according to five factors (risk of bias, heterogeneity, imprecision, indirectness, and publication bias). The 35 new or modified evidence profiles provided new absolute effect estimates due to altered baseline risks or relative effects. These new absolute estimates resulted in six recommendations changing direction and four changing in strength and in the addition of seven new recommendations. 4Often, one or more interventions or comparators were excluded due to feasibility issues (eg, medications not being readily available or infrequently used in Norway and equal options being suggested instead). 5Five recommendations were modified due to new baseline documentation from a Norwegian or Scandinavian population. Twenty-six recommendations were modified partly due to updated documentation published after November 2010. For the chapters on major orthopedic surgery and pregnancy, the relative effect of low-molecular-weight heparin vs placebo as thromboprophylaxis was derived from a meta-analysis by Collins et al cited in the AT9 chapter on the prevention of VTE in nonorthopedic surgical patients.25 6Baseline risk refers to the risk in untreated patients. 7Disagreement with how the balance between the benefits and harms of an intervention is valued resulting from disagreement with the panel’s assumption of typical patient preferences and values. 8Disagreement with the assessment of the overall quality of the evidence across individual outcomes. 9Disagreement with the assessment of resource use, cost-effectiveness, and so forth. 10Eight recommendations were modified from strong to weak, whereas the remaining 11 were modified from weak to strong. 11A PICO element is excluded, added, or modified, for example, exclusion of intermittent pneumatic compression device as one of several thromboprophylactic options, adding apixaban to dabigatran as a treatment option in patients with atrial fibrillation, and modifying where AT9 states vitamin K antagonists to warfarin in the adapted guideline because this is the only available vitamin K antagonist in Norway. See Figure 1 and 2 legends for expansion of abbreviations.Grahic Jump Location

Tables

References

Guyatt GH, Norris SL, Schulman S, et al. Methodology for the development of antithrombotic therapy and prevention of thrombosis guidelines: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):53S–-70S.
 
Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. [CrossRef] [PubMed]
 
Guyatt GH, Oxman AD, Kunz R, et al; GRADE Working Group. Going from evidence to recommendations. BMJ. 2008;336(7652):1049-1051. [CrossRef] [PubMed]
 
Goldet G, Howick J. Understanding GRADE: an introduction. J Evid Based Med. 2013;6(1):50-54. [CrossRef] [PubMed]
 
Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):1-72.
 
Fervers B, Burgers JS, Haugh MC, et al. Adaptation of clinical guidelines: literature review and proposition for a framework and procedure. Int J Qual Health Care. 2006;18(3):167-176. [CrossRef] [PubMed]
 
Harrison MB, Graham ID, van den Hoek J, Dogherty EJ, Carley ME, Angus V. Guideline adaptation and implementation planning: a prospective observational study. Implement Sci. 2013;8:49. [CrossRef] [PubMed]
 
ADAPTE Collaboration. The ADAPTE Process: Resource Toolkit for Guideline Adaptation. Version 2.0. Guideline International Network website. www.g-i-n.net. Accessed November 2013.
 
Treweek S, Oxman AD, Alderson P, et al; DECIDE Consortium. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. [CrossRef] [PubMed]
 
Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381-389. [CrossRef] [PubMed]
 
Kristiansen A, Brandt L, Agoritsas T, et al. Applying new strategies for the national adaptation, updating, and dissemination of trustworthy guidelines: results from the Norwegian adaptation of the antithrombotic therapy and the prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2014;146(3):735-761.
 
Agoritsas T, Kristiansen A, Agarwal A, et al. Updating and monitoring of clinical practice guidelines: performance of McMaster Premium LiteratUre Service (PLUS) in identifying new key studies. Chest. In press. doi:10.1378/chest.14-1366.
 
Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. What is “quality of evidence” and why is it important to clinicians? BMJ. 2008;336(7651):995-998. [CrossRef] [PubMed]
 
Guyatt GH, Akl EA, Crowther M, Gutterman DD, Schuünemann HJ; for the American College of Chest Physicians Antithrombotic Therapy and Prevention of Thrombosis Panel. Executive summary: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):7S–-47S.
 
Dick F, Ricco JB, Davies AH, et al. Chapter VI: follow-up after revascularisation. Eur J Vasc Endovasc Surg. 2011;42(suppl 2):S75-S90. [CrossRef] [PubMed]
 
Hamm CW, Bassand J-P, Agewall S, et al; ESC Committee for Practice Guidelines. ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation: the Task Force for the Management of Acute Coronary Syndromes (ACS) in Patients Presenting Without Persistent ST-Segment Elevation of the European Society of Cardiology (ESC). Eur Heart J. 2011;32(23):2999-3054. [CrossRef] [PubMed]
 
Gaglia MA Jr, Waksman R. Systematic review of thienopyridine discontinuation and its impact upon clinical outcomes. Eur Heart J. 2011;32(19):2358-2364. [CrossRef] [PubMed]
 
The Norwegian Directorate of Health. Nasjonale Retningslinjer for individuell primærforebygging av hjerte- og karsykdommer. Oslo, Norway: The Norwegian Directorate of Health; 2009.
 
The Norwegian Directorate of Health. Behandling og rehabilitering ved hjerneslag. Oslo, Norway: The Norwegian Directorate of Health; 2010.
 
Debourdeau P, Farge D, Beckers M, et al. International clinical practice guidelines for the treatment and prophylaxis of thrombosis associated with central venous catheters in patients with cancer. J Thromb Haemost. 2013;11(1):71-80. [CrossRef] [PubMed]
 
Farge D, Debourdeau P, Beckers M, et al. International clinical practice guidelines for the treatment and prophylaxis of venous thromboembolism in patients with cancer. J Thromb Haemost. 2013;11(1):56-70. [CrossRef] [PubMed]
 
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P; Board of Trustees of the Guidelines International Network. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525-531. [CrossRef] [PubMed]
 
Lenzer J. Why we can’t trust clinical guidelines. BMJ. 2013;346:f3830. [CrossRef] [PubMed]
 
Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews. J Clin Epidemiol. 2012;65(1):62-72. [CrossRef] [PubMed]
 
Gould MK, Garcia DA, Wren SM, et al. Prevention of VTE in nonorthopedic surgical patients: antithrombotic therapy and prevention of thrombosis, 9th ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2012;141(2_suppl):e227S-e277S. [PubMed]
 
MAGIC. Public guidelines. MAGIC app website. http://www.magicapp.org/public. Accessed December 2013.
 
MAGIC. DYNAM-IT. MAGIC website. http://www.magicproject.org/dynam-it. Accessed December 2013.
 
Graham R, Mancher M, Wolman DM. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press; 2011.
 
Shiffman RN, Dixon J, Brandt C, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;5:23. [CrossRef] [PubMed]
 
Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58. [PubMed]
 
Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172(21):1628-1633. [CrossRef] [PubMed]
 
Silagy CA, Weller DP, Lapsley H, Middleton P, Shelby-James T, Fazekas B. The effectiveness of local adaptation of nationally produced clinical practice guidelines. Fam Pract. 2002;19(3):223-230. [CrossRef] [PubMed]
 
Burgers JS, Anzueto A, Black PN, et al; ATS/ERS Ad Hoc Committee on Integrating and Coordinating Efforts in COPD Guideline Development. Adaptation, evaluation, and updating of guidelines: article 14 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):304-310. [CrossRef] [PubMed]
 
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Find Similar Articles
CHEST Journal Articles
PubMed Articles
  • CHEST Journal
    Print ISSN: 0012-3692
    Online ISSN: 1931-3543