Correspondence to: Martin J. Tobin, MD, Division of Pulmonary and Critical Care Medicine, Loyola University of Chicago Stritch School of Medicine, Edward J. Hines, Jr. Veterans Administration Hospital, Fifth Ave and Roosevelt Rd (111N), Hines, IL 60141; e-mail: firstname.lastname@example.org
Drs. Karanicolas, Kunz, and Guyatt say the first principle supporting the scientific base of evidence-based medicine (EBM) is systematic summaries. But being “systematic” is far less important than avoiding errors. EBM advocates1write that compliance with criteria for “systematic summaries” will ensure “the whole truth.” The EBM metaanalysis2on weaning indexes, however, contains > 15 methodologic errors.3 Yet it is absolutely systematic.1
Their second principle is need for “a sophisticated hierarchy of evidence.” “Sophistication” is neither a necessary nor sufficient condition in science. What counts is soundness. EBM provides neither (experimental) testing of its grading system, nor cogent arguments for why EBM is able to overcome a problem that defeated some of the best brains of the twentieth century (the logical positivists). A discussion of revised grading systems is given in Table 1
Their third principle is clinical decisions should take into account patient preferences (in addition to evidence). For the life of me, I cannot see how this provides a scientific base to EBM. Patient preferences have always transcended scientific considerations. We do not transfuse sentient Jehovah Witnesses. How can EBM add “the patient dislikes metered-dose inhalers” to “double-blind randomized design” in their grading system? The two are incommensurate. It is attempting to square the circle.
The authors use Figure 1A to assert that metaanalysis provides a means for early discovery of certain truth. LeLorier et al4compared 19 metaanalyses against 12 subsequent randomized controlled trials (RCTs) on the same topics. Metaanalysis failed to accurately predict RCT outcome 35% of the time. Villar et al5 found much the same. Moreover, metaanalyses often contain serious errors: > 15 in that on weaning.2–3
The authors assert that examples in Table 1 validate EBM grading. But not one of these studies was undertaken with the aim of testing the validity of EBM grading. Instead, Table 1 simply shows that conclusions from one study are often refuted by subsequent studies. This is hardly earth shattering; it reflects the cumulative, superseding character of science. The authors imply that errors of studies undertaken before RCTs get corrected once a high-quality RCT is undertaken. If true, why have a series of RCTs on steroids and insulin in the ICU produced somersaulting conclusions? The steroid-insulin experience is nothing surprising. It simply reflects that science is a murky process of groping in the dark. It is misleading to suggest that EBM grading provides a shortcut: acting like a fractionating column, eliminating errors and providing a distillate of incorrigible knowledge.
The fourth paragraph under “A Hierarchy of Evidence” gives five criteria for adjusting grading ratings.6 Compare these with the eight requirements for reliable research in my Table 3. The stark contrast emphasizes that EBM is built on statistics. Statistics, however, do not undo systematic errors or breaches of internal validity. Statistics are not the fount of new penicillins. Statistics do not confer the wisdom needed to make the right decision for a particular patient (my Fig 1).
At a metaphysical level, the EBM dream is reminiscent of Marx's well-intentioned hypothesis: regulation of society based on scientism will guarantee human happiness. The EBM version: clinical practice based on grading of clinical research studies will result in wiser decisions. Utopian projects aimed at eradicating uncertainty and introducing universal good have produced more misery than good fortune.7
Dr. Tobin Professor of Medicine and Director, Division of Pulmonary and Critical Care Medicine, Loyola University of Chicago Stritch School of Medicine.
The author has no conflict of interest to disclose.
PEEP = positive end-expiratory pressure; Fio2 = fraction of inspired oxygen.
From Brower RG, Lanken PN, MacIntyre N, et al, NIH-NHLBI ARDS Network. Higher vs lower positive end-expiration pressures in patients with the acute respiratory distress syndrome. N Engl J Med 2004; 351:327–336
Become a CHEST member and receive a FREE subscription as a benefit of membership.
Individuals can purchase this article on ScienceDirect.
Individuals can purchase a subscription to the journal.
Individuals can purchase a subscription to the journal or buy individual articles.
Learn more about membership or Purchase a Full Subscription.
Institutional access is now available through ScienceDirect and can be purchased at myelsevier.com.
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 2
Customize your page view by dragging & repositioning the boxes below.
Enter your username and email address. We'll send you a reminder to the email address on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.