Tuesday, April 22, 2008

How Much a Paper Submission Costs?

I have been reading the post by Lance Fortnow about the cost of a class, and what is the amount that students pay collectively for an hour of teaching. This made me think of a similar calculation for the cost of submitting a paper to a conference. We are accustomed to submit papers and then asking for high-level reviews, often disregarding the associated costs. "What cost?", you will ask, given that everything in academic reviewing is done in a gratis, voluntarily basis. Fundamentally our peer reviewing system is based on an implicit tit-for-tat agreement: "I will contribute a number of reviews as a reviewer, so that others can then review my own papers".

In most cases, though, some employer is paying the reviewer (a university, a research lab...) and reviewing consumes some productive time. A typical computer scientist with PhD will have a salary above $100K per year, which roughly corresponds to a $50/hr-$100/hr salary. A typical review (at least for me) takes at least 3 hours to complete, in the best case, corresponding to a cost of $150 to $300 per review. Additionally, every paper submission gets 3-4 reviewers, which results in a cost per submission of $500 to $1000 per paper. Therefore, a conference like SIGMOD, WWW, KDD, and so on, with 500-1000 submissions per year, consumes from $250,000 to $1,000,000 in resources, just for conducting the reviewing. I simply find that amount impressive.

This leads to the next question: Have you ever thought about your balance? How many papers do you review and how many papers do you submit per year? If someone had to pay $1000, I doubt that we would see many half-baked submissions. Or, if credit was given for each conducted review, then we would have more reviewing resources available. I do not advocate a system based on monetary awards, but before complaining about the quality of the reviews that you get, think: What is your balance?

Friday, April 18, 2008

Journal of Database Management Research (JDMR)

A common discussion in any CS community is the hardness/unfairness/randomness of having a paper accepted in one of the highly selective conferences. Given the low acceptance rate (say 15%) and assuming a normal distribution of quality, it is normal to have a large number of papers that are roughly equivalent in terms of quality, but end up in different sides of the acceptance threshold. This binary decision threshold leads to rejections that are perceived as unfair and to acceptances that are perceived as random. Furthermore, it is often the case that a paper is rejected, the authors fix the comments of the reviewers, they resubmit the paper to another conference, and the (new set of) reviewers identify other problems to reject the paper.

One way to change this is to allow for a journal-like process, in which the papers are submitted, returned to the authors with comments, the paper is fixed, and then published. We already have traces of this process in SIGMOD and VLDB, where the authors can reply to the concerns of the reviewers but this process only helps for misunderstandings and not for fixing significant parts of the paper.

One initiative in the database community that tries to address this issue is the proposal for a new journal named Journal of Database Management Research (JDMR). The goal of JDMR is to gradually replace the current reviewing process for the database conferences. The details of the proposal are still under discussion so I will not list the whole proposal here, but the basic elements of the proposal are the following:
  • Instead of having a PC for each conference, there will be a single Review Board that will be reviewing papers year-round. The expectation is to have a significant turnaround every year.
  • Rapid refereeing, trying to reach the speed of reviewing in the life sciences which is significantly lower than in our community.
  • Papers will go through rounds of revision and when the concerns of the reviewers will be addressed, the paper will be published to the JDMR.
  • The conferences (SIGMOD, VLDB, etc) will pick which of the published papers from JDMR will be presented.
  • Rejected papers will not be allowed to be resubmitted in their existing form for a period of one year.
Overall, I think that the proposal makes a lot of sense. However, there are some issues that need to be examined more closely. For example, the prohibition to resubmit rejected papers is a double-edged sword. Yes, the authors will have the incentive to avoid sending half-baked papers but, on the other hand, "noisy rejections" by reviewers that did not understand/read carefully/appreciate the paper will have a very significant negative effect: in practice it may make the community more insular as papers that do not make it the first time will be essentially expelled from the database conferences. Perhaps an appeal process with limits can be used to allow authors to ask for a new set of reviewers, just like in tennis the players have a limited number of contests for the referee decisions.

Another issue that needs to be examined is the relation with existing journals (TODS, VLDBJ, TKDE): given that the papers for JDMR will be now journal publications but of shorter length, what is the role of the other journals?

Anyway, this is an interesting development and I expect this to improve both the internal and the external perception of the publication process in the database community. Myself, I do not see any major problems except for the couple of issues mentioned above. Do you see any other issues?

Update (7/27/2008): The website of JDMR is now live and the vision statement together with the transition plans are now (officially) posted.

Saturday, April 5, 2008

The Importance of "Everything Else Being Equal"

Lately, we have been working on a project for improving local search for hotels. The project, funded by a Microsoft Virtual Earth award, tries to incorporate into the ranking function features such as the price of the hotel, its amenities, geographical characteristics such as "proximity to a beach" (inferred using image classification of satellite images), and customer reviews. The final outcome is a ranking function that we call "value for the money".

Of course, the most important thing is to actually test how good the final ranking is. For this reason, we generated our ranking for a few US cities, and compared it against some other baselines, such as "rank by distance", "rank by price", "rank by Tripadvisor ranking" and so on. To give some guidance, we gave the corresponding title to each ranking and presented them to the users, asking them to choose the ranking that they preferred most.

Overwhelmingly, the users picked out ranking, that was titled "Value for the Money". The difference was so striking, that we got suspicious. Therefore, we decided to run the following experiment. We swapped the titles of the rankings and we named a baseline technique as "Value for the Money". Interestingly enough, the baseline technique, which was the worst performing one, ended up being the most preferred under the new title! In other words, we simply gave a fancy title to a bad ranking technique and the users were convinced that this was the best possible ranking!

Finally, we decided to run a truly blind test. We presented pairs of ranked hotel lists to the users, without any title, and asked the users to pick the one that they liked best. Our ranking strongly outperformed the other baselines, which I guess is a good thing :-). But I think that the important lesson from this experiment was to really compare apples to apples. Add even subtle "non-equal" external elements and the experiment can give deceiving results.