Saint Matthews Strikes Again : an Agent-Based Model of Peer Review and the Scientific Community Structure

This paper investigates the impact of referee reliability on the quality and efficiency of peer
review. We modeled peer review as a process based on knowledge asymmetries and subject
to evaluation bias. We tested various levels of referee reliability and different mechanisms
of reviewing effort distribution among agents. We also tested different scientific community
structures (cohesive vs. parochial) and competitive science environments (high vs. low
competition). We found that referee behavior drastically affects peer review and an equal
distribution of the reviewing effort is beneficial only if the scientific community is homogeneous
and referee reliability is the rule. We also found that the Matthew effect in the
allocation of resources and credit is inherent to a ‘winner takes all’ well functioning science
system, more than a consequence of evaluation bias.