Any binary decision that depends on some explicit or implicit threshold will always be problematic. Whatever is threshold+epsilon gets in, whatever is threshold-espilon is out. An epsilon difference generates significantly different outcomes. To make things worse, the area around the threshold is typically densely populated. No matter where we put the threshold, papers with small differences in quality, even under perfect quality assesment, get very different treatments.
Here is an alternative. Allow all authors to decide whether to publish their papers or not. With one condition. They will also publish together the reviews for the paper. The paper got 3 strong rejects and the authors still want to publish the paper? Fine!
If such a system was in place, then most of the authors would seek to get good reviews instead of trying to pass the threshold and get into the publish-land.
Going a step further? Keep the reviewed versions of the paper together with reviews of earlier versions. Did the authors address the comments? They can have a statement describing that. Later reviewers can take a look and see whether this is the case. This policy would also encourage submissions of only high-quality results.
This can also be matched with the requirement for the reviews to come from at least some high quality reviewers, but I will leave this for another post.
