Stop Publishing!
The last few months, I feel that I have an endless queue of reviewing tasks to complete. WWW, followed by DBRank, followed by EC, followed by KDD, followed by VLDB, followed by WebDB, plus an NSF panel, plus some journal reviews, and I have rejected invitations for a few additional conferences including SIGIR, SIGMOD, and a few others. This puts my count at least 40 reviews over the last 4-5 months. (Just to break even, I will need to submit 10-12 papers.)
Needless to say, having such a reviewing load means that I cannot really do a good job in reviewing. My reviews have been declining in quality, signalling that I need to learn to say no.
At the same time, I also notice that the other reviews that are being submitted are not that great either. While on the one hand I feel happy ("OK, I am not that bad"), on the other hand I feel that this cannot be good. If nobody has time to review thoroughly, what is the whole point of peer reviewing?
One solution is to accept fewer invitations for PCs, allowing for more time per paper. However, I know that without volunteering time and effort for reviewing the system cannot work! There are simply not that many reviewers available!
Part of this problem is, of course, the increased need to get papers published: For tenure, for getting a job, even for being admitted to a PhD program! I feel that there is something wrong when, to be admitted to a PhD program, you need to have already research experience. This increased need for more and more publications, overloads the reviewing system which unfortunately has a limited capacity.
Unfortunately, it is not easy to reverse this trend. The incentives are setup in a way to encourage quantity of publications, preferably in good venues. Once the paper gets accepted in a good venue, the goal is achieved. This encourages publications that are "good enough" to pass the reviewing process, not papers that have stellar quality. And with the increased noise in the reviewing process, the distinction between "good enough to be published" and "what the hell, send it, we may get lucky" is getting blurrier and blurrier. In fact, I have cases in my papers that the reviewers did such a poor job that I never understood at the end if my paper was worth getting published, or I got just lucky.
I noticed though a positive development! Through the Greek University Reform Forum, I learned that:
The German Research Society (DFG) has introduced new guidelines for applications and evaluations of proposals, which will be valid as of July 1, 2010.
A rough-and-ready translation of the main points:
- Applicants should cite in their CV only up to FIVE publications, those which are most relevant for the proposal at hand;
- In reports about running projects, a maximum of TWO publications PER YEAR. In case of projects with more than one PIs, a maximum of THREE publications PER YEAR.
- The goal of the new guideleines is to put emphasis on quality instead of quantity and to stop the flood of publishing for the sake of the numbers.
It has caused quite some stir here in Germany and the voices to enforce such rules also for decisions on faculty positions are getting louder.
While some of these ideas are already in place (e.g., NSF also allows only five publications in the CV), the idea of "counting" only two publications per year for each project is definitely a step towards the right direction. It is not going to be trivial to reverse the "get as many publications in top venues as possible" trend, but every step towards de-emphasizing quantity counts.
After all, Pollock was also getting paid by the piece when he worked for the Federal Art Project, but none of his famous paintings come from that period.
The idea of limiting quantity to achieve a better measure of quality is being tried in some places in the US. I think some schools in UIUC have adopted similar guidelines (if informally).
ReplyDeleteOne of the advantages of quantity over quality in a heterogeneous field like HCI is that quality is in the eye of the beholder, and not everyone (on a hiring committee or a tenure review committee) will have the same notion of what constitutes quality research. With quantity, you have a higher chance of producing work that someone will like.
Also, there is something suspicious to me about quotas applied to a noisy signal. You stand a good chance of losing discriminatory power if you set the quota too low.
Having said that, I am in general in favor of higher-quality rather than higher quantity for the same reasons you describe. Perhaps there is a better way to achieve quality reviewing (which may reduce the quantity of published papers!). My current favorite approach is some kind of scheme that allows reviewers to be rated, and quality reviewing to be recognized for the effort that it is. Here is one example.
I want to focus on your first paragraph: Once I had a professor that told me that organizations seek "good" members for their stuff in order to get represented in forums, standard groups and similar activities. Unfortunately this results in have good workers work for the organization at roughly 30% of their work time (no citation on this of course). So it seems the case with academic members that accept PC inclusion in far too many places. It is good for someone to be included in as many places yearly. It builds a (networking) brand which is different (and sometimes easier) from the brand name that establishes itself from the actual publications.
ReplyDeleteI think part of the solution is a consensus-driven rule that would not allow one to be a member of say 3 PCs per year. That way there would be room for others, on the verge of volunteering, to step up. You were in at least 7 PCs, doing the work of 2+ people. How long can you keep up at this pace?
@adamo May I guess that the person that made the comment about representation in professional bodies was an electrical engineer? In places where professional bodies have influence beyond the narrow academic boundaries, I can see the validity argument.
ReplyDeleteBut for the narrow scope of conference PCs, I doubt that anyone cares if I serve in 2, 3 or 30 PCs per year.
Note that I already decline ~50% of the invitations. I would have no problem saying no to even more invitations. But at the same time, someone has to review all these papers. Since I contribute workload to the system, I should also serve to lessen the load, right?
Yes, I can decline 80% of the invitations and encourage the widening of the pool of reviewers. I believe, though, that we already cast a very wide net. Should we reach the point of having PhD students writing paper reviews, just to expand the pool of reviewers? This is not a solution in my mind. We cannot expect a student with 3-4 years of research experience to have the breadth and depth required to judge the novelty and correctness of a research paper.
At the same time, I agree with you, how long can I keep up reviewing like crazy? (I cannot, and I know that I am not doing a good job!) This is the reason that I think that official measures that discourage LPU's (least publishable units) are a correct step towards improving the current status.
Panos, well-said. But here are maybe some questions to add to the debate:
ReplyDelete1. What is quality? Is it only a paper of high methodological quality, or it includes a path-breaking research question and insight? The former is always needed, but the latter I think happens not frequently, by definition.
2. What is the VALUE created through publishing, and how this value is measured? Let's say you have a market of widgets (or cars). The producers of high quality cars create value, but this requires high-cost (skills, effort, time etc) and the output is expensive so very few consumers can buy it. In this market, low quality producers create A HUGE AMOUNT OF VALUE, because they create affordable cars that will satisfy the needs of many consumers. So remove the LOW QUALITY, and what you will achieve is a huge loss of social welfare.
Now I am not saying that publishing is the same kind of market, but it could be that "lower-quality" research that satisfies niches creates significant value. In any case the few top journals have limited capacity by definition. Maybe you could call that the "long-tail of publishing"?
In fact, using the analogy of cars' market, it seems that publishing is a peculiar type of SELF-REFERENTIAL market, i.e. the producers of papers are also "consumers" of them and no outside consumer is asked about the value (maybe I am wrong). In that scenario, the producers can very well collude: creating many kinds of journals so that everybody publishes, and in the end no-one goes out of "business". That would suggest that "outsiders" need to be asked more about the value created, and maybe other changes...