Some thoughts on Area Chairing / Meta-Reviewing

Thanks to Alexis Palmer, Casey Kennington and Nathan Schneider for their comments on an earlier version of this.

PREAMBLE: Why I Am Writing This.

This year, I was asked for the first time to be an Area Chair both for ACL and EMNLP. It was a great insight and experience. First of all, I would like to state that I am totally convinced that the vast majority of ACs takes this job very seriously and does a great job. However, in my opinion, there is room for improvement in the instructions to ACs, which would be especially helpful for first-timers like myself this year. Luckily, I was working with really awesome Senior ACs in each case who were very responsive, encouraging and helpful during my process in creating my own recipe for how to do a good area chairing job. Here, I’d like to share my experience. Maybe it will be helpful for someone in the future.

What are the Responsibilities of an Area Chair / Meta-Reviewer?

In the ACL community, this role is responsible for looking through the initial reviews (usually three), making a recommendation whether the paper is ready to be published, and writing a short so-called meta-review summarizing the major points of the reviews as well as the reviewers’ discussion.

POINT 1: The Timing.

It’s a great honor to be asked to be an AC. It’s great for your CV. It gives you lots of great insights. If you’re asked, it means you have reached some kind of senior level in you research area. So, if you’re asked, by all means, support the community and say yes. IF you can make the time. Because it really is time-intensive, even if you know the subject well. The tasks were, with a realistic assumption how long it takes (at least the first time):

Checking potential problems in reviewer assignment, around 10 minutes per paper. I had almost 20 papers one time, so this took me around 2 hours. No problem this far.

As the reviews and author responses come in, you are supposed to start a discussion. This means actually reading all reviews thoroughly and starting a discussion among reviewers, at least in those cases where you’d like to have further opinions or want reviewers to elaborate / discuss their points. Count minimum half an hour per paper on average, which means … block an entire day in your calendar, just to be safe.

Soon afterwards, you have to write the meta-reviews. Luckily, some papers had been withdrawn at this point. Still, 14 meta-reviews to write. Of course, you don’t have to provide an advisor’s view on each not-yet-ready paper. However, behind many, if not most papers, there are PhD students who spent a lot of time on their papers. The least these young people entering our community deserve is a polite and useful feedback on why their paper was not accepted/needs an other iteration of research or writing. Remember, they might review your work one day. More thoughts on how to achieve this below. The point of this list was … right, timing. Above, I called myself a new senior-level person in my research area, however, to be honest, there are still many things that I need to look up. Sometimes, for example, I know that some previous paper did something but does this really mean that the paper under review is not novel? And so on. Plus, in a few cases, you won’t be able to rely on the initial reviews in that case. If you are already more senior than me, you might not have this problem. Well, I wanted to do a good job so I looked up some things here and there. Just to be clear, in one case I ended up recommending a paper whose merit the initial reviewers did not see, but to convince the SAC and the PCs, I actually did dig a bit deeper and almost wrote a review myself. I think it is just the case that these days many reviewers are relatively new to the field and not all initial reviews are to be counted upon. But hey, this is why we are there, and we shouldn’t just say “back luck with the reviewers,” but instead be the safety net in that case. Also, this doesn’t mean you have to read all 20 papers, but in that example I started to wonder from just reading the abstract and the reviews. So, if a case is a clear accept or reject, you might be done in an hour. But for some cases, I really spent two hours or more. In sum, I think I spent around 4-5 days writing meta-reviews, including my entire weekend. If you’re more senior than me, you might be faster, but reading the abstract, reviews, author response, discussion, and writing the review takes at least 45 minutes per paper. (Apparently Nathan is faster. :))

Two take-aways: (1) If you agree to be an AC, block some days in your calendar early. (2) Conference organizers (note: ARR may have alleviated some things here), please allow more time for thorough meta-reviewing in the future, e.g., two weeks would be helpful.

From the SAC point of view, it is extremely helpful if you communicate in a timely fashion. Even if it’s to say “I haven’t gotten to this yet but I will tomorrow.” When coordinating amongst 15 ACs (for example) it would be tedious if half of them didn’t respond to an email and needed reminding. (Thanks to Nathan Schneider for this point.)

POINT 2. The Recipe for Writing a Meta-Review.

As an author, it is important to me to notice that the AC has actually gone through the relevant material and arrives at a recommendation for the right reasons. Any decision above your level will most likely happen just based on your scores and text. (Note: Do not explicitly state your recommendation in the text, though, because the PC decision may differ from yours.) This is my suggested structure for meta-reviews. Of course, some types of papers may need a slightly different structure and I diverged from my scheme if I didn’t have to say anything important for some categories. It’s more like a template to be modified.

SUMMARY. 2-3 sentences (but no more) summarizing what the paper is about, as usual.

STRENGTHS and CONTRIBUTIONS. Here, I am stating the reasons to accept the paper. I am not repeating the contributions list from the paper, instead, I am just noting the points that I would consider real contributions.

WEAKNESSES. Things to be improved, especially for rejected papers. For instance, a methodological flaw or a missed comparison with particular prior work. (Keep in mind what has already been published at the submission deadline of your conference, though.)

WRITING. Some comments on whether the writing is clear enough or needs improvement.

REVIEWS and DISCUSSION. Summarize what the reviewers recommend, especially what happened in the discussion. Make sure to keep this double-blind. Possibly also add in information that was received via the author response.

SUMMARY. A summary of your evaluation of the paper, e.g., “needs only minor modifications” or “would profit from another iteration.” As stated above, do not make your recommendation explicit here, though.

ADDITIONAL COMMENTS. In cases where you disagree with the initial reviews, especially when recommending rejection or when agreeing only with some of the reviewers, state some points that make clear that you took a closer look at the work.

ACs have more time than SACs to evaluate the reviews and papers. Hence, do not to be afraid to take sides in weighing the pros and cons for papers with mixed reviews. It is super helpful for SACs if the meta-review has comments like “one reviewer asked for X, but in my view this is beyond the scope of the paper” or “reviewers complained about some points that were unclear but they are minor details and should be easy to clarify for the camera-ready.” (Again thanks to Nathan Schneider for this point.)

POINT 3. Final Notes.

Thanks for reading up to here. If you made it here, I am sure you will be a great AC because you are interested in this subject. Please let me know what you think about these ideas. Finally, I hope you will have many great insights and interactions with the community while AC’ing (meta-reviewing)!