**(pdf) Abstract
**1. Despite the wide application of meta-analysis in ecology, some of the traditional

methods used for meta-analysis may not perform well given the type of data characteristic

of ecological meta-analyses.

2. We reviewed published meta-analyses on the ecological impacts of global climate

change, evaluating the number of replicates used in the primary studies (ni) and

the number of studies or records (k) that were aggregated to calculate a mean effect

size. We used the results of the review in a simulation experiment to assess

the performance of conventional frequentist and Bayesian meta-analysis methods

for estimating a mean effect size and its uncertainty interval.

3. Our literature review showed that ni and k were highly variable, distributions were

right-skewed and were generally small (median ni = 5, median k = 44). Our simulations

show that the choice of method for calculating uncertainty intervals was

critical for obtaining appropriate coverage (close to the nominal value of 0.95).

When k was low (<40), 95% coverage was achieved by a confidence interval (CI)

based on the t d istribution t hat uses a n a djusted s tandard e rror (the H artung–

Knapp–Sidik–Jonkman, HKSJ), or by a Bayesian credible interval, whereas bootstrap

or z distribution CIs had lower coverage. Despite the importance of the

method to calculate the uncertainty interval, 39% of the meta-analyses reviewed

did not report the method used, and of the 61% that did, 94% used a potentially

problematic method, which may be a consequence of software defaults.

4. In general, for a simple random-effects meta-analysis, the performance of the

best frequentist and Bayesian methods was similar for the same combinations of

factors (k and mean replication), though the Bayesian approach had higher than

nominal (>95%) coverage for the mean effect when k was very low (k < 15). Our

literature review suggests that many meta-analyses that used z distribution or

bootstrapping CIs may have overestimated the statistical significance of their results

when the number of studies was low; more appropriate methods need to be

adopted in ecological meta-analyses.