Heterogeneity in effect size estimates
Abstract
In conducting empirical research in the social sciences, the results of testing the same hypothesis can vary depending on the population sampled, the study design, and the analysis. Such variation, referred to as heterogeneity, limits the generalizability of published scientific findings. We estimate heterogeneity based on 86 published meta-scientific studies. In our data, the estimated design and analytical heterogeneity are substantive and of at least the same magnitude as sampling uncertainty, whereas population heterogeneity is smaller. The results suggest low generalizability of the empirical findings under consideration across different study designs and statistical analyses.
- Publication:
-
Proceedings of the National Academy of Science
- Pub Date:
- July 2024
- DOI:
- Bibcode:
- 2024PNAS..12103490H