Until you’ve served on a program committee, the review process may seem opaque and mysterious.  PC members generally try to do a good, impartial job of selecting papers, but PC’s are human institutions, humans are flawed, so PC’s are imperfect.   Understanding the PC process can make the results less confusing, and provide useful guidance for crafting papers.

The Conference Review Process

The details vary from conference to conference and year to year, but here are the basic steps.

First, the PC chair assigns your paper to a small subset (3-5) of the PC for review based on conflicts of interest (e.g., professors from your school can’t review your paper), the topics you list for your paper, and the reviewers’ interests.  Hopefully, there are enough reviewers that 1) you aren’t conflicted with and 2) have the expertise to review your paper.  If not, you get what you get.  The reviewers may include member of an “external review committee” who are not on the PC proper and don’t attend the meeting.

The reviewers read and review your paper, provide some numeric scores, and some written feedback.  Reviewers usually review between 10 and 20 papers.  That’s between 100 and 300 pages of reading.  Reviewers have a couple months to do their reviews, and some spread them out.  Others do them all at the minute.  If they do the reviews early, the papers won’t be fresh in their minds.  If they do them all at the end, they are rushed.

With so much reading to do and so many reviews to write, PC members are prone to looking for reasons to reject a paper.  Poor writing, missing citations to the reviewer’s favorite papers, and questionable methodology are all sometimes-spurious reasons for rejection.

The PC meets for 1-2 days (usually, it seems, in a stuffy, windowless room) to discuss the papers.

The roadmap for the discussion is the “discussion order” created by the PC chair before the meeting.  The papers are ordered by their average numeric score, highest to lowest.

Broadly, there are three classes of papers in discussion order:

• Clear accepts — The reviewers loved them, so they will almost certainly get in after a brief discussion.
• Needs discussion — Could go either way, see below.
• Clear rejects — Not discussed, not getting in.

The PC spends most of its time on the middle group.  The discussion will go something like this:   One of the reviewers is designated as the “discussion lead.”  They will summarize the reviews orally.  Then, each of the reviewers has a chance to offer some additional comments.   If there’s consensus among the reviewers for accept or reject and no one else on the PC objects, the fate of the paper is settled.

Otherwise, a broader discussion ensues with reviewers and non-reviewers asking questions, posing opinions, stomping their metaphorical feet, etc.  After awhile (there’s usually a time limit), the whole PC will vote.

A few things to note:  First, reviewers on the ERC are not present, so their opinions may not carry as much weight as the other reviewers.  It’s the discussion lead’s job to advocate for their position.  They may or may not be effective.

In the final vote, PC members vote on the paper even if they haven’t read it, so they are voting based on the discussion.  A vocal champion or a vocal detractor can have outsized effect on the paper’s fate.  Unfortunately, it is easier for a detractor to sink a paper than it is for a champion to save it.

Lessons

There are (at least) two important lessons in the above discussion:

First, the process is imperfect and a little bit random.  Good papers can and do get rejected, even if they have some strong reviews.  So, don’t take your rejections too hard.

Second, being on PC is a lot of hard work and a lot of reading.  PC members often have little patience for unclear papers, so make sure yours is well-written.

Why Not Some Other Way?

There is a lot of general dissatisfaction with the process, and people have proposed and/or tried alternative models.  If you want to hear about them, just sit down with some professors in the bar at a conference and ask how they would fix the reviewing process.  You’ll get plenty of contradictory suggestions.