Avenues for critical discourse at influential scientific journals: A cross-disciplinary assessment of post-publication peer review policies and practice

Critical scrutiny of published scientific research, often called post-publication peer review (PPPR), can highlight errors, limitations, or alternative interpretations that were not adequately addressed during traditional peer review. Currently, there is little empirical data documenting how academic journals handle this important aspect of scientific self-correction. We conducted a systematic and cross-disciplinary assessment of PPPR policies and practice amongst 330 influential journals operating across 22 scientific disciplines. 123 (37%) journals did not offer any options for submitting PPPR. 207 (63%) journals offered PPPR options such as letters, commentaries, or web comments, but often imposed limits on length (median 1000 words) and time-to-submit (median 12 weeks). The most restrictive limits were 175 words and 2 weeks. Amongst PPPR-offering journals, we found that 1.9% (95% CI [1.4, 2.6]) of a random sample of 2066 research articles published in 2018 were the subject of at least one PPPR. A detailed examination of 58 PPPRs found, for example, that the majority (76%) received an author reply. The majority of those replies (93%) asserted that the authors’ original claims were unchanged. Clinical Medicine stood out as the domain with the most active PPPR culture, with journals most frequently providing PPPR options and publishing the most PPPR, but also imposing the strictest length and time-to-submit limits. Overall, our findings suggest that influential academic journals often pose serious barriers to the cultivation, documentation, and dissemination of PPPR. We propose several policy changes that journals could adopt to promote a richer culture of post-publication critique. Keywords: peer review, post-publication peer review, letter to the editor, meta-research, journal policy, self-correction, scientific criticism