Perhaps this lack of recognition is one of the reasons that we actually have bad reviews. Once someone is a member of a program committee, there are very few incentives to do an excellent job. The only real incentive is to do an average job and not be embarrassed during the discussion phase or when the program chair (or the associated editor) reads the reviews. There is almost no incentive to do an excellent job. At best, the authors and the program chair will be happy with the quality of the reviews and there will give thanks to the anonymous reviewer. It is a pretty thankless job!
One way to avoid this is to start a reputation system for reviewers. When the reviews are ready, the text of the review is sent to the authors. The authors then rate the quality of the reviews, and the reviewer is assigned (privately) a reviewing score. The highest rated reviewers are then recognized for their efforts.
There are a couple of issues of course. Authors will be happy when the reviewer is positive towards the paper and this gives the incentive to the reviewer to be lenient and propose acceptance for many papers. To counter-balance this, we should condition the reviews on the score assigned by the reviewer to the paper. Therefore, the reviewer evaluation will be a matrix like:
| Reviewer's score for the paper | ||||||
| Reject | Weak Reject | Weak Accept | Accept | Strong Accept | ||
| Author's Evaluation of Review | Not Useful | 1 | 1 | 1 | 1 | 0 |
| Useful | 1 | 2 | 2 | 2 | 1 | |
| Very Useful | 0 | 3 | 1 | 1 | 0 | |
where the number in the cell correspond to the number of reviews with the given ratings. For example, the values in the column "Weak Reject" mean that the reviewer submitted 1+2+3=6 reviews with a "Weak Reject" rating and 3 of them were deemed "Very Useful" by the authors, 2 of them were deemed "Useful" and 1 was rated as "Not Useful".
Potentially, such a metric can be propagated from conference to conference, and the reviewer can even make the matrix public if desired.
It is important though, not to see this approach as a punitive measure. Having "halls of shame" and other such tactics may result in a backslash. "Why participate in a program committee and risk seeing my name published in a hall-of-shame list?" The number of researchers available to review is not growing fast in contrast to the number of publications that are being submitted to the conferences. The goal of the system should be to reward and encourage participation, not punish.
Wouldn't it be nice to allow PhD students to review papers, and encourage them to review papers not with the goal of rejecting them but with the goal of offering constructive feedback? And a recognition like "top reviewer for ACM 2008 conference on..." would definitely look good on the CV and signal that the student can give high quality feedback.
Update: Of course, you should also look at the (imho, classic): "Reviewing the Reviewers" by Ken Church.
