Affiliations: Charleston, SC
Dr. Heffner is Professor of Medicine, Department of Medicine, Medical University of South Carolina.
Correspondence to: John E. Heffner, MD, FCCP, Division of Pulmonary and Critical Care Medicine, Medical University of South Carolina, 96 Jonathan Lucas St, PO Box 250623, Charleston, SC 29425; e-mail: firstname.lastname@example.org
I have developed a fondness for meta-analysis because of
the jeopardy we physicians face in thinking we know more about some
subjects than we actually do. With 33,000 new citations added to
MEDLINE each month, it is no surprise that published reports of new
clinical approaches stick in our minds and get entrenched into practice
before sufficient evidence establishes their merit. Periodically, a
good dose of meta-analysis purges our misperceptions and allows us to
see the strength of the evidence—or lack thereof—that underlies our
clinical habits. Meta-analysis fulfills its primary role by aggregating
data from conflicting or underpowered studies in an attempt to
establish statistical evidence of the value of an
intervention.1 But to my way of thinking, meta-analysis
provides a more important secondary benefit of critically appraising
the quality of the data entered into its review. By detecting
weaknesses of primary studies and identifying what we don’t know, but
think we do, meta-analysis sets research agendas and keeps us from
practicing “no-evidence”-based medicine.
I am delighted, therefore, that Freeman and colleagues have published
in this edition of CHEST (see page 1412) a meta-analysis
examining the relative risks in critically ill patients of percutaneous
dilatational tracheotomy (PDT) by the Ciaglia technique2
compared to the standard surgical procedure. PDT has emerged within a
short 15 years to become the principle method in many ICUs for placing
a surgical airway. The technique has spawned a wealth of published case
series, reviews, and conflicting editorial opinions but surprisingly
few adequately designed, prospective, comparative studies examining its
value relative to the standard technique. So what can we learn about
PDT from this meta-analysis review?
First, we should look at how the study fulfilled the primary goal of
meta-analysis in establishing statistically significant differences
between competing interventions. The investigators observed
similarities between PDT and surgical tracheotomy in three of the eight
measured outcomes, but noted shorter operative times (9.84-min
shorter), less postoperative and perioperative bleeding, less overall
postoperative complications, and less postoperative stomal infections
with PDT. Although nearly all of the complications favoring PDT
appeared to be of minor clinical importance, we should critically
appraise the meta-analysis itself before accepting PDT as the“
procedure of choice.”
Meta-analyses are most useful when they analyze aggregate data from
high quality, randomized controlled trials (RCTs). As indicated by the
authors, two of the five studies were nonrandomized. Inclusion of
nonrandomized studies is risky in a meta-analysis because they usually
overestimate the average effect of a treatment.3 Of the
RCTs, one used an inadequate randomization technique (alternate weeks),
and the randomization method was not mentioned in another. In the
absence of appropriate subject allocation, I can’t help but suspect
that patients assigned to standard tracheotomy were more critically
ill, more technically challenging for airway placement, or affected by
some other confounders. Consequently, the primary studies may have
biased the results toward more complications in the standard
tracheotomy group. This concern is especially relevant considering that
none of the primary studies provided measures of disease severity.
Meta-analyses can provide an estimate of the impact of selection bias
by analyzing data for effect size separately for the randomized and
nonrandomized studies, which was not done by Freeman and colleagues. It
should be noted also that the Mantel-Haenszel fixed effect model used
in this analysis may produce extremely misleading conclusions when
confounders are not taken into account in the initial design of primary
As an additional standard, meta-analyses should pool data from studies
that are homogeneous.4) Homogeneous studies treat
patients with similar interventions and measure similar outcomes.
Meta-analyses that aggregate heterogeneous studies are pooling data
from apples and oranges. The resulting conclusions are of dubious value
except for concocting fruit cocktails. I am concerned that the five
studies in the present analysis may have been too heterogeneous to
allow pooling of their data. For instance, one study quantifies
bleeding in milliliters and another by the ability to control blood
loss with digital pressure. Even within the same study, different
definitions of bleeding were used between tracheotomy groups.
Heterogeneity of outcomes because of differing definitions is the only
reason I can see that one of the studies observed stomal infections
with standard tracheotomy in 0% of patients and another study in 63%;
one study observed overall complications in 8% of patients, and
another study noted complications in 87%. Freeman and colleagues
established homogeneity of the primary studies with the Q
statistic. Tests of homogeneity, however, have low power; test results
that do not reject the hypothesis that primary studies are homogeneous
and do not prove that the same outcomes are being
As their secondary purpose, meta-analyses should examine the quality of
the primary studies and identify weaknesses in study design that could
contribute to overgeneralized results. I would like to know, for
example, how many of the primary studies used data collectors blinded
to study purpose and group assignment, excluded from enrollment
technically challenging patients (short necks, obesity), or experienced
high dropout rates. Considering the subjective nature of many of the
outcomes measured, lack of blinding is an important consideration.
Without a measure of study quality, the suspicious reader might suspect
that the zeal of the primary investigators for a new procedure could
inflate estimates of the value of PDT. By not reporting quality
criteria, I assume Freeman and colleagues found little evidence of
rigorous study designs. My review of the primary studies found no use
So what have we learned from this meta-analysis on the relative merits
of PDT in our ICUs? I believe Freeman and colleagues have clearly
demonstrated our unclear knowledge of the short-term risks of
tracheotomy. As they discuss, we know even less about the long-term
effects of PDT on the airway or the relative costs of the two
procedures. Previous studies state that PDT has a cost advantage
because it can be performed in the ICU rather than the operating
room.5–6 In one of the primary studies, however, both
procedures were done in the operating room,7and in
another both were done in the ICU.8 When performed in
similar settings, large discrepancies in cost largely
I believe Freeman and colleagues have done us a great service by
establishing a research need for rigorously designed, randomized trials
that examine standardized outcomes from tracheotomy. Considering how
often we make decisions about modes of airway support, we should
recognize—and quickly respond to—what we have now learned we don’t
know about these commonly performed interventions.
But what do we do in the meantime in selecting between standard
tracheotomy and PDT? Although insufficient data establish that PDT is
safer than the standard procedure in general populations of critically
ill patients, Freeman and associates adequately support that it is just
as safe in skilled hands when clinically important end points are
considered. Application of PDT should depend on local expertise and
practice patterns within an ICU until better outcome data emerge. But
should PDT be profiled as “simple to learn and perform,” as stated
in the meta-analysis introduction? Both PDT and standard surgical
tracheotomy present considerable risks for acute and long-term
complications. Their use for critically ill patients requires extensive
procedural training, airway management skills, quality monitoring, and
respect for the potential lethality of each of these invasive
procedures. As taught to me by an old surgeon, there is no such thing
as “simple” surgery. In the final analysis, this is one thing that
we do know.
Become a CHEST member and receive a FREE subscription as a benefit of membership.
Individuals can purchase this article on ScienceDirect.
Individuals can purchase a subscription to the journal.
Individuals can purchase a subscription to the journal or buy individual articles.
Learn more about membership or Purchase a Full Subscription.
Institutional access is now available through ScienceDirect and can be purchased at myelsevier.com.
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Web of Science® Times Cited: 8
Customize your page view by dragging & repositioning the boxes below.
Enter your username and email address. We'll send you a reminder to the email address on record.
Athens and Shibboleth are access management services that provide single sign-on to protected resources. They replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Athens or Shibboleth authentication, please contact your site administrator to receive your user name and password.