مقاله سال ۲۰۱۶

Determining the relative merit of a specific academic contribution is often a daunting task. Science has developed institutions – like academic journals – as well as accompanying routines – like peer review – to systematically address this task. Today these institutions play a dominant and influential role even though past research successfully documented their limitations: peer review, for instance, is known to lack robustness (Gans and Shepherd, 1994) as well as objectivity (Bedeian, 2003) andmight give rise to shrewed incentives (Macdonald and Kam, 2010; Day, 2015). Citation metrics are similarly contested, as they suffers from a general bias due to the skewed distribution of academic attention (Solla-Price, 1965), entail substantial problems of internal validity (because citation counting measures relative impact as a proxy for quality; Amin and Mabe, 2000), incorporate a series of conceptual biases (Kapeller, 2010) and induce reactive behavior among authors, reviewers and editors (Reedijk and Moed, 2006). In this paper we take a different perspective on evaluation in academia. We will assume that these problems simply do not exist and that evaluation procedures are valid, transparent, fair and as objective as possible: we aim to explore the properties of scientific discourse under the assumption that two main evaluative instruments in science – peer review and journal rankings – function rather objectively.

In assessing this question we employ a simple model simulating the academic production process: scientists produce manuscripts of different quality and try to publish these papers in journals. Journals, conversely, try to select those manuscripts with the highest quality for publication and rely on inputs from referees to make that judgement. The model is kept as simple as possible. Our main contribution is to show that even under overly optimistic assumptions, where decisions and journal-rankings are completely objective and distorting factors like opportunistic behavior or academic feuds are absent, the underlying structure of scientific publishing will inevitably exhibit idiosyncratic properties. Specifically, we show that scientific publishing can only be efficientin an idealized setting, where authors, referees and journals are perfectly objective and accurate. This idealized scenario turns outto be extremely unstable and already a tiny amount of noise fundamentally alters basic properties ofthe academic production process for the worse.We present our model in Section 2 and discuss the main results in Section 3. Summarizing, ourpaper aims todemonstrate thatthe way scientific publishing operates will give rise to a series ofinteresting and partly unexpected phenomena; some of these phenomena are harmful to scientific progress despite the best intentions of authors, editors andreferees and,hence,provide anadditionalpoint ofdeparture for a critical assessment of the inner routines of academic institutions.