Peer Reviewing for Oral Presentations?
Everyone who has attended a conference knows that the quality of the talks is very uneven. There are talks that are highly engaging, entertaining, and describe nicely the research challenges and solutions. And there are talks that are a waste of time. Either the presenter cannot present clearly, or the presented content is impossible to digest within the time frame of the presentation.
So, my question is: Why don't we have peer reviewing for oral presentations?
We already have reviewing for the written part. The program committee examines the quality of the written paper and vouch for its technical content. However, by looking at a paper it is impossible to know how nicely it can be presented. Perhaps the seemingly solid but boring paper can be a very entertaining presentation. Or an excellent paper may be written by a horrible presenter.
Why not having a second round of reviewing, where the authors of accepted papers submit their presentations (slides and a YouTube video) for presentation to the conference. The paper will be accepted and be included in the proceedings anyway but having a paper does not mean that the author gets a slot for an oral presentation.
Under an oral presentation peer review, a committee looks at the presentation, votes on accept/reject and potentially provides feedback to the presenter. The best presentations get a slot on the conference program. This also allows the conference to accept more papers that are worthy of inclusion to the proceedings, without worrying about capacity constraints.
Why not having a second round of reviewing, where the authors of accepted papers submit their presentations (slides and a YouTube video) for presentation to the conference. The paper will be accepted and be included in the proceedings anyway but having a paper does not mean that the author gets a slot for an oral presentation.
Under an oral presentation peer review, a committee looks at the presentation, votes on accept/reject and potentially provides feedback to the presenter. The best presentations get a slot on the conference program. This also allows the conference to accept more papers that are worthy of inclusion to the proceedings, without worrying about capacity constraints.
Some other side benefits of this scheme:
- Presentations are accessible in an archival format
- Authors have hard incentives to be better presenters
- The time of the attendees in conferences is not wasted in clearly sub-par presentations.
And if someone says that this system is biased towards good and sleek presenters, I would argue that the system is already biased towards good authors. A well-written paper will eventually have a higher impact than one that is badly written. Same thing for presentations.
Learning to communicate properly the results of our research should be a goal, not an afterthought.
Learning to communicate properly the results of our research should be a goal, not an afterthought.
that would be ideal, but I don't think it would work.
ReplyDeleteFirst it would be unfair if we had the following situation: a high quality paper written by a bad quality presenter, his idea wouldn't be exposed and discussed within the community during the conference.
Second it would demand a tima, from both authors and reviewers, to prepare, record, upload, review and all those process that already demand an inexistent time during the paper writing process.
For the first question, I think that a bad presentation actually damages the work. In the same vein, we could argue that the system is also unfair for brilliant technical researchers that do not know how to write.
ReplyDeleteFor the second part, all authors of accepted papers now have to present and prepare the material. This process just requires the material to be presented earlier and submitted for review.
This is a great idea in theory, and I would love to see this implemented. However, the biggest problem is going to be the committee itself. I have seen varying opinions about presentation, and unlike papers, it's lot more subjective. Many people who write great papers, also seem to present badly (either due to lack of time or interest).
ReplyDeleteAnother problem is: are you going to reject a paper (or not invite the author), because the presentation is bad? Will the committee have to look at updated presentations? Who has time for that, in the age of over loaded PCs
It's fine (maybe) if you can show that authoring a "non-presented" paper gets you as many citations...but I suspect that's not the case. And if the people who aren't presenting don't get as many citations, this system is a form of discrimination (non-native speakers, people with speech impediments, etc.). Also, as a professor, as many times as I get my students to practice a talk before they go to a conference, there's nothing like actually giving it in front of the public (for good or bad). By denying them that experience/practice, you're denying them the ability to improve.
ReplyDeleteI don;t think that you can judge objectively based only on the slides, simply because a presentation consists of two things: the slides and what the presenter says. In fact, what the presenter says is far more important than what the slides show. The slides augment the oral part and depend on it. In most conferences, things are other way around: the slides are important and the presenters just reads them out loud. Look for example at the excellent use of slides in Larry Lessig's presentation http://www.youtube.com/watch?v=7Q25-S7jzgs
ReplyDeleteIf you would receive these slides, could you make sense of it? I doubt it...and yet the presentation style -in my opinion- belongs into text books. If you want get rid of bad presentations, i think another idea might be more effective: ban bullet lists from slides!
Also, some conferences have actually awards for the best presentation.
@Manolis I suggested to submit slides *and* a video. Not slides alone.
ReplyDelete@Anon 2:01pm:
"if you can show that authoring a "non-presented" paper gets you as many citations":
Why would I want to show that? I want people to be good presenters, not to opt for the "no-presentation" option by submitting horrible presentations.
"this system is a form of discrimination (non-native speakers, people with speech impediments, etc.)"
We already have this discrimination built-in. People that are non-native speakers cannot write papers in English as easily as the English speakers. Plus, I do not care to fix this. I want well-written and well-presented papers in a language that most researchers can understand (English). If this means hat non-native speakers need to work harder in order to write and speak nicely, let it be.
"By denying them that experience/practice, you're denying them the ability to improve."
If they present nicely, this system does not deny them anything. If they are bad, back to the drawing board, and practice more. Give internal talks, go to less demanding workshops to learn to present.
It is identical with writing papers. The more papers they write, the better they become. The process includes both acceptances and rejections, and both are useful learning experiences.
@Pradeep:
"Many people who write great papers, also seem to present badly (either due to lack of time or interest)"
Good. Let them write good papers. We will read them later. We do not have to suffer through their horrible presentations. If they want to present, they can put the time to prepare properly instead of wasting the time of 100 people in the audience.
"Another problem is: are you going to reject a paper (or not invite the author), because the presentation is bad?"
No, the paper will be in the proceedings but the author does not present. Maybe they can have a poster if they want to interact with the audience. (KDD asks all authors to have posters)
"Who has time for that, in the age of over loaded PCs"
I sympathize, I am victim of the overloading myself. But this should be an easier task: (a) The papers are already vetted for technical interest, (b) Judging a presentation is more intuitive than having to read the paper, (c) we can always do it in a simply rating scheme like for YouTube videos and for Amazon reviews.
The goal is not to reject presentations. The goal is to have the incentives in place for people to be better presenters.
Ultimately, I would like to see 100% of the presentations being accepted.
I think the idea is great, mostly due to the availability of video version of the paper by the author himself, however, practically, this could easily be achieved by a well organized and equipped conference (see for example WIN conference 2009).
ReplyDeleteNevertheless, another problem is in the small details. I think that the lack of easy means of recording high quality video using home equipment would present a barrier for authors (sometimes simple video editing skills / the appropriate software can make wonders to a presentation).
A simple tool that allows video editing & secure publishing of the presentations would definitely help here (would you like your presentation out on YouTube before the conference and even after when people know it wasn't accepted...)
Hi Pano, I suggest you look at Toastmasters International for a formula for giving and receiving oral and written feedback on presentations.
ReplyDeleteToastmasters clubs help people to practice and improve their public speaking skills in a fun and safe environment.
I think its a great idea and very practically implementable!
ReplyDeleteThe only question is when do the presentation videos/slides have to be submitted? Right now people are used to working until a few minutes before the talk tweaking the slides. Perhaps a month in advance will give enough time for review (So effectively there would be 2 rounds of review-acceptance separated by a couple of months).
Another question is whether all authors will have access to good audio-visual equipment to record their talk. For example right now I don't have access to such equipment, but I suppose I could get it if it becomes necessary. I guess just a basic camcorder would do.
Apart from better presentations, this will also remove the artificial barrier of accepting only as many papers as presentation slots which seems like a good idea in the age of electronic proceedings.
I think if a major conference adopted this proposal it could really catch on!
@Mugizi: Yes, timing seems to be the only real issue with this proposal. I think this model will work best with the VLDB mode of operations, in which papers get accepted throughout the year.
ReplyDeleteAlthough not implemented, the vision of VLDB is to disconnect publication from presentation: The presentations will be a selected set from all the papers accepted over the last year. In this setting, reviewing presentations can work well, without pressing time constraints.
I do believe that a basic level of quality control can work wonders. Even the fact of knowing that there will be *some* filtering can force many researchers to pay attention in the presentation and improve their presentation skills.
Extending the idea a bit:
ReplyDelete* Accepted authors may submit presentation if they want extended or "featured" speaking slot.
* Best presentations get extended/better speaking slot (say 25 minutes or, say, plenary slots). others get 5 minutes / madness slots / small session slots.
Benefits: as above
Cost: smaller (e.g., fewer items to review).
I am not rejecting the idea, but let me put another one on the table:
ReplyDeleteYou could have some sort of evaluation/rating of a presentation given in a conference by its audience. After some time and accumulation, you could have, for future conferences, some a priori knowledge of who is a good presenter and who is not, which you could use to decide which papers will be presented orally. And of course this could be extended to research groups, schools and so on...
Needless to say I strongly believe that the problem is always with the presenter, not with the content of the specific presentation.
I think something that is very real is that some topics cannot possibly be presented well in 20 minutes or even 45 minutes to an audience who does not already know everything around that subject. Many subjects can, but some cannot. This suggestion would remove the pressure to present such subjects anyway to an inappropriate audience. I think that is marvelous!
ReplyDelete