Replication in R
R code that regenerates the numbers this paper reports, starting from the dataset published on this site. It reproduces 24 of 30. The 6 that do not match are listed below.
Run it
Rscript run.R
run.R is the whole package. Save it and run it: it reads the dataset from
this site if the CSV is not sitting beside it, and loads the shared conventions file the same way,
so it works on its own in an empty directory. It needs R with fixest, and depending on
the paper lme4, metafor, plm, BMS or
LowRankQP. It writes results.json, one value per number, each named for
where it appears in the paper.
How it compares with Stata
Most of these papers were estimated in Stata, and the two programs differ in places that change
printed digits. Those conventions are stated once, in
stata_compat.R, and shared by every replication on this site:
ivreg2's large-sample variance, SSC winsor's order statistics,
xtreg's treatment of singleton groups, and the restricted-ML default of
xtmixed, which is not the default of the mixed command that replaced it.
Numbers that do not match the printed paper
Cells are named as they are in results.json. The reason for each difference follows the table.
| cell | paper | this code |
|---|---|---|
| Panel A: OLS SE (publication selection) coef | -1.016 | -1.016527 |
| Panel A: OLS Constant (true effect) t | 1.69 | 1.605356 |
| Panel A: OLS Constant (true effect) p | 0.099 | 0.108939 |
| Panel A: IV SE (publication selection) coef | -1.234 | -1.234917 |
| Panel A: IV Constant (true effect) coef | 0.038 | 0.038692 |
| Panel B: Mixed Constant (effect beyond bias) z | 1.43 | -1.427388 |
Table 3's note says the standard errors are clustered at the study level. The published Panel A was estimated without clustering: heteroskedasticity-robust standard errors reproduce every printed t-statistic exactly, and the authors' log of the published run carries no cluster option. This package reports both. Clustered by study the coefficients are unchanged and the inference is weaker: the standard-error term gives t = -1.73, p = 0.091 by OLS and t = -1.76 by IV, against printed t of -5.18 and -4.81, and the constant gives t = 0.87, p = 0.388 against a printed 1.69 and 0.099. Both sets are in results.json. The printed values are left exactly as published.
The printed z carries no sign. It must be negative: the coefficient is -0.133, the p-value is 0.153, and every other cell in the table is signed. The authors' log gives -1.43. The printed value is left as published.
Files
- run.R, the replication
- stata_compat.R, the Stata conventions
- targets.json, the numbers as printed in the paper, taken from the paper rather than from the code
- results.json, the numbers this code produced
- REPLICATION_STATUS.md, the full comparison, generated from this package's own output