Do outlier treatment decisions matter in meta-analysis?

25 July 2026  ·  Zuzana Irsova, Tomas Havranek, Martina Luskova, T. D. Stanley  ·  meta-analysis methods

First published on LinkedIn, 25 July 2026. Archived with the rest of our writing at /komentare/posts/2026-07-25-do-outlier-decisions-matter/.

So, how much do different outlier treatments matter for meta-analysis results?

We have just preprinted a study that recomputes 358 behavioral science meta-analyses under five different outlier treatments.

The mean effect barely moves: the median absolute change in Cohen’s d is at most 0.047 and usually much smaller. But the interpretation moves more. In 11.5% of the meta-analyses, at least one treatment changes statistical significance; in 15.9%, it changes whether the effect reaches a smallest effect size of interest.

Winsorizing changes conclusions least often; DFBETAS changes them most often.

Paper, online appendix, and data: https://meta-analysis.cz/outliers Pre-registration: https://doi.org/10.17605/OSF.IO/97CMV Replication package: https://doi.org/10.5281/zenodo.21216506

Joint work with Tomas Havranek, Martina Lušková, and T. D. Stanley

Title page of the working paper Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses by Tomas Havranek, Zuzana Irsova, Martina Luskova and T. D. Stanley, dated July 2026, with the full abstract below the title.
Table 3 of the paper, Changes in statistical significance and smallest effect size of interest. Two panels compare four outlier treatments (drop-extreme, absolute studentized residual above 3, Winsorize 5/95, and absolute DFBETAS above 2 over the square root of k) against the do-nothing baseline, each under random-effects and unrestricted weighted least squares estimators. Winsorizing changes the fewest conclusions and DFBETAS the most.

All notes  ·  More meta from Prague