Monday, May 25, 2009

Evaluation Feedback and Stakhanovist Research Profiles

Every year, after the Spring semester, we receive a report with our annual evaluation, together with feedback and advice for career improvement (some written, some verbal). Part of the feedback that I received this year:
  1. You get too many best paper awards, and you do not have that many journal papers. You may want to write more journal papers instead of spending so much time polishing the conference papers that you send out.
  2. You are a member of too many program committees. You may consider reviewing less and write more journal papers instead.
I guess that having a Stakhanovist research profile (see the corresponding ACM articles) is a virtue after all.

(*) Alexey Stakhanov was a miner in Soviet Union, who cut 102 tons of coal during a six-hour shift with a pneumatic drill, when the average production being 6-7 tons. Stakhanov's record created the Stakhanovite movement where workers were encouraged to exceeded production targets, typically trying to overcome previous production records.

Monday, May 18, 2009

Public Paper Reviews

I am now trying to review a journal paper. Unfortunately, the paper is completely unreadable,  being a long list of definitions, "lemmas," and "theorems" (the quotes are intentional). Even more unfortunate is the fact that the author does not give any background but rather cites some recently published paper of his, which contain all the necessary background. 

Trying to understand the paper, I attempted to read the published paper. Well, no luck! The paper was also horribly written. I started wondering who in his right mind decided to accept this paper to an ACM journal, and what the reviewers were saying. I simply cannot believe that anyone in his right mind would actually read such papers, and even try to write any meaningful review afterwards. Most probably the reviewers gave up and agreed after many revisions to allow the paper to be published, hoping that they will never have to read the paper again. Or may be I am wrong and the paper is indeed a hidden gem?

So, how can we avoid such cases? Here is my not so original suggestion: Publish the (anonymous) reviews together with each paper! 

I cannot see anything negative with that. It will make everyone happier. People that write high-quality reviews would not mind seeing their names being published together with the reviews. Other reviewers will see what is a high-quality review and hopefully will try to imitate the style. 

In fact, the practice of publishing a commentary for each paper is not new. I have seen many papers in statistics being published with eponymous commentary. Often, reading the reviews is more interesting than reading the paper.

Furthermore, the reviews will offer a quick overview of the contributions and shortcomings of the paper. It will also allow the reader to understand what lead to the acceptance of the paper. Was it a new idea? An excellent experimental evaluation? Or just the reviewers could not even read the paper and just gave up, giving a lukewarm "accept"?