Abstract

Meta-analysts routinely face estimates that look too large or extreme. Yet, how to handle them is left to the reviewer's judgment. The methods for detecting such estimates are well known. What is missing is an informed assessment of how much alternative handling choices might change a meta-analysis' conclusions. We fill this gap by analyzing the effects of four pre-registered handling treatments across 358 behavioral science meta-analyses with at least ten estimates. Each outlier handling treatment is estimated by two estimators (random effects and unrestricted weighted least squares), and compared to the 'do-nothing' baseline on three outcomes: the pooled effect, statistical significance, and whether the effect reaches the smallest effect size of interest (|d| ≥ 0.20). Alternative outlier handling treatments have little effect on the meta-analysis mean as the median absolute change in Cohen's d is at most 0.047 and often much less. Yet, at least one of these four treatments in combination with one of these estimators reverses the statistical significance of 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%. Winsorizing has the least effect and DFBETAS the most.

How much each treatment moves the result, against doing nothing — pooled over the two estimators (Tables 2 and 3 of the paper). The abstract’s 0.047 is the largest median shift under a single estimator, DFBETAS under random effects.

Treatment Median absolute change in d Significance changes SESOI changes
Drop the most extreme estimate 0.024 3.35% 5.87%
|Studentized residual| > 3 0.006 2.80% 5.73%
Winsorize at 5/95 0.006 2.38% 1.82%
|DFBETAS| > 2/√k 0.045 5.45% 7.12%
At least one of the four — 7.7% 10.3%

The unit is a meta-analysis-by-estimator cell. The last row uses the 715 cells estimable under all five treatments; each treatment row uses all cells estimable under that treatment (715 or 716). Counting a meta-analysis as changed if either estimator changes, the rates in the last row rise to 11.5% for statistical significance and 15.9% for SESOI. Median absolute change in d is pooled over both estimators. SESOI is the smallest effect size of interest, |d| ≥ 0.20.

Materials: the study was pre-registered before the treatments were applied to the frozen dataset, and the replication package (data, R code, and outputs) is archived on Zenodo.

Reference: Tomas Havranek, Zuzana Irsova, Martina Luskova, T. D. Stanley (2026), “Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses.” Charles University, Prague. Available at meta-analysis.cz/outliers.

Read the paper

Headline result

Sensitivity of meta-analysis results to outlier handling: the methodological finding is median shift at most 0.047 (Cohen's d); significance flips in 11.5% of cases (across four pre-registered outlier treatments and two estimators), based on 358 behavioral science meta-analyses (Havranek et al. 2026).

How to cite

Tomas Havranek, Zuzana Irsova, Martina Luskova, T. D. Stanley (2026), "Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses." Charles University, Prague. Available at meta-analysis.cz/outliers.

BibTeX 
@misc{havranek2026outliers,
  author  = {Tomas Havranek and Zuzana Irsova and Martina Luskova and T. D. Stanley},
  title   = {Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses},
  year    = {2026},
}