(pdf)  Abstract
1. Despite the wide application of meta-analysis in ecology, some of the traditional
methods used for meta-analysis may not perform well given the type of data characteristic
of ecological meta-analyses.
2. We reviewed published meta-analyses on the ecological impacts of global climate
change, evaluating the number of replicates used in the primary studies (ni) and
the number of studies or records (k) that were aggregated to calculate a mean effect
size. We used the results of the review in a simulation experiment to assess
the performance of conventional frequentist and Bayesian meta-analysis methods
for estimating a mean effect size and its uncertainty interval.
3. Our literature review showed that ni and k were highly variable, distributions were
right-skewed and were generally small (median ni = 5, median k = 44). Our simulations
show that the choice of method for calculating uncertainty intervals was
critical for obtaining appropriate coverage (close to the nominal value of 0.95).
When k was low (<40), 95% coverage was achieved by a confidence interval (CI)
based on the t d istribution t hat uses a n a djusted s tandard e rror (the H artung–
Knapp–Sidik–Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap
or z distribution CIs had lower coverage. Despite the importance of the
method to calculate the uncertainty interval, 39% of the meta-analyses reviewed
did not report the method used, and of the 61% that did, 94% used a potentially
problematic method, which may be a consequence of software defaults.
4. In general, for a simple random-effects meta-analysis, the performance of the
best frequentist and Bayesian methods was similar for the same combinations of
factors (k and mean replication), though the Bayesian approach had higher than
nominal (>95%) coverage for the mean effect when k was very low (k < 15). Our
literature review suggests that many meta-analyses that used z distribution or
bootstrapping CIs may have overestimated the statistical significance of their results
when the number of studies was low; more appropriate methods need to be
adopted in ecological meta-analyses.