Aug 10, 2021
Disclaimer: this rant is the opinion of Ben Stock and him alone.
My colleague Andreas Zeller recently noted that reviewing in the security community isn’t great. Let me share my thoughts on why this might be.
How do different fields review papers? In 2021, I served on the PCs of @PLDI (programming languages), @ICSEconf (software engineering), and @acm_ccs (security). How do papers, reviews, and processes compare? And which dos and don'ts did I learn? Enjoy! https://t.co/ksOjqWIRCA— Andreas Zeller (@AndreasZeller) July 29, 2021
Since 2018, I have served on the PCs for USENIX&CCS four times, NDSS three times, and Oakland twice. In total, I have written (sometimes with the help of my students) over 200 reviews for those conferences alone. Arguably, this has been a service to the community.
For me, “serving” on a PC goes beyond community service. There are plenty of reasons why I want to be on a PC. (Naturally, this list is very biased to my perspective, but I am sure that others will have similar motives)
As well-intentioned as the community service of being on a PC might be, late, short, and non-constructive reviews are rather a disservice to the community. Moreover, it is disrespectful towards authors (who get sub-par feedback), other PC members (who spend much time on constructive reviews and discussions), and PC chairs (who need to literally beg people to be on time and engage in discussions). I understand that we all have other duties (at CISPA, luckily not as many): writing papers, teaching courses, and writing proposals.
If I am late in writing a paper, I can’t submit it. If I am late in preparing a lecture, I will be embarrassed for being unprepared. If I am late handing in a proposal, I won’t get funding. If I am late in submitting my reviews … I get increasingly begging emails to please get things done. Maybe our community should add the same consequences to late reviews: can’t submit your review (excluding justified cases such as sickness) on time? You get kicked out of the PC. Or, for each paper that you are late on, one of your papers gets auto-rejected.
This obviously is overdoing it a bit, but we should stop treating PC memberships purely as a service that we do for the community for free and acknowledge that PC members benefit from this privilege and should treat it as equally important as the rest of our duties.
Take what you will from my rant. I don’t want to downplay the importance of volunteers to help papers become the best they can be. But we should change our attitude with respect to expectations: nobody will kindly remind you and beg you to be on time for writing a high-quality paper, submitting an excellent proposal, or teaching an award-winning course. Why should it be different when submitting a review or engaging in a discussion?