- Lies, damned lies and statistics: The adverse incentive effects of the publication bias
- Number of pages
- Amsterdam: Faculteit Economie en Bedrijfskunde
- Document type
- Working paper
- Faculty of Economics and Business (FEB)
- Amsterdam School of Economics Research Institute (ASE-RI)
In this project I investigate the use and misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I use a data set of over 135 thousand p values from more than five thousand papers. I inspect (1) the way in which the p values are reported and (2) their distribution. The main findings are following: first, the authors choose the mode of reporting their results in an arbitrary and strategic way, that is, to make them seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values "just above" significance thresholds directly, whereas other values are reported by means of inequalities (e.g. "p<.1"), they round the p values down more eagerly than up and choose between the significance thresholds and between one- and two-sided tests after seeing the data. Further, about 8% of reported p values are inconsistent with their underlying statistics (e.g. F or t) and it appears that there are "too many" "just significant" values, suggesting use of data or model manipulation techniques to bring the p value to the right side of the threshold.
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library, or send a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.