Wednesday, March 14, 2012

When do reviewers submit their reviews? (ACM EC 2012 version)

A few weeks back, just after the deadline for the submission of papers for ACM EC'12, I wrote a brief blog post, showing how more than 60% of the submissions came within the last 24 hours before the deadline.

Now we are in the process of reviewing the papers, and the deadline for reviewers to submit their reviews was on March 5th, a few days before sending the reviews back to the authors for feedback. Here is the plot of the submission activity. On the x-axis we have the time, and on the y-axis the percent or reviews received by that time. With a yellow line, I marked the official deadline.

The similarities with the submission dynamics for papers are striking. One day before the deadline, we had received only 40% of the reviews. Within the next 24 hours, we jumped from 40% to 85%, receiving approximately 300 reviews during that period. If we go to 36 hours before the deadline, we can see a jump from 20% to 85%.

The key difference with the paper submissions plot is that reviewers can submit late, much to the chagrin of the PC Chairs and the Senior PC members that are trying to get the discussion going. You can see clearly that, after the deadline, we needed one additional day to go from 85% to 90%, and then another extra day to reach the 98% completion rate.

On the positive note, despite the love of both authors and reviewers to submit material very close to the deadline, the overall quality of the submissions for EC'12 seems to be pretty high. (Self-selection at work, I guess.) With Kevin, we are doing our best to see how we can accommodate as many papers as possible.