Meta-analysts routinely face estimates that look too large or extreme. Yet, how to handle them is left to the reviewer's judgment. The methods for detecting such estimates are well known. What is missing is an informed assessment of how much alternative handling choices might change a meta-analysis' conclusions. We fill this gap by analyzing the effects of four pre-registered handling treatments across 358 behavioral science meta-analyses with at least ten estimates. Each outlier handling treatment is estimated by two estimators (random effects and unrestricted weighted least squares), and compared to the 'do-nothing' baseline on three outcomes: the pooled effect, statistical significance, and whether the effect reaches the smallest effect size of interest (d=0.20). Alternative outlier handling treatments have little effect on the meta-analysis mean as the median absolute change in Cohen's d is at most 0.045 and often much less. Yet, at least one of these four treatments in combination with one of these estimators reverses the statistical significance of 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%. Winsorizing has the least effect and DFBETAS the most.

Materials: the study was pre-registered before the treatments were applied to the frozen corpus; the online appendix is on OSF; and the replication package (data, R code, and outputs) is archived on Zenodo.


Reference: Tomas Havranek, Zuzana Irsova, Martina Luskova, T. D. Stanley (2026), “How much does outlier and influence handling change meta-analytic conclusions? Evidence from 358 behavioral science meta-analyses.” Charles University, Prague. Available at meta-analysis.cz/outliers.