Disclaimer: this rant is the opinion of Ben Stock and him alone.

My colleague Andreas Zeller recently noted that reviewing in the security community isn’t great. Let me share my thoughts on why this might be.

How do different fields review papers? In 2021, I served on the PCs of @PLDI (programming languages), @ICSEconf (software engineering), and @acm_ccs (security). How do papers, reviews, and processes compare? And which dos and don'ts did I learn? Enjoy! https://t.co/ksOjqWIRCA

— Andreas Zeller (@AndreasZeller) July 29, 2021

Is reviewing only a community service?

Since 2018, I have served on the PCs for USENIX&CCS four times, NDSS three times, and Oakland twice. In total, I have written (sometimes with the help of my students) over 200 reviews for those conferences alone. Arguably, this has been a service to the community.

For me, “serving” on a PC goes beyond community service. There are plenty of reasons why I want to be on a PC. (Naturally, this list is very biased to my perspective, but I am sure that others will have similar motives)

1. I was often frustrated with the lack of constructive criticism and can do my part to improve that (still arguably to the benefit of the community).
2. I know way before others what is happening in the community (up to 6 months in the old cycles).
3. I know what makes a good submission rather than a good final version, and I also know what PCs like in submissions. Before, this was very intransparent, except for the reviews for my own papers.
4. I need to be on the PCs. I am on tenure track, and being part of PCs shows that I am valued enough by the community to judge other researchers’ works.
5. being on PCs allows me to have more weight in nominating my own students as reviewers, which is not only beneficial to their careers but also my TT evaluation.

As well-intentioned as the community service of being on a PC might be, late, short, and non-constructive reviews are rather a disservice to the community. Moreover, it is disrespectful towards authors (who get sub-par feedback), other PC members (who spend much time on constructive reviews and discussions), and PC chairs (who need to literally beg people to be on time and engage in discussions). I understand that we all have other duties (at CISPA, luckily not as many): writing papers, teaching courses, and writing proposals.

Consequence of being late

If I am late in writing a paper, I can’t submit it. If I am late in preparing a lecture, I will be embarrassed for being unprepared. If I am late handing in a proposal, I won’t get funding. If I am late in submitting my reviews … I get increasingly begging emails to please get things done. Maybe our community should add the same consequences to late reviews: can’t submit your review (excluding justified cases such as sickness) on time? You get kicked out of the PC. Or, for each paper that you are late on, one of your papers gets auto-rejected.

This obviously is overdoing it a bit, but we should stop treating PC memberships purely as a service that we do for the community for free and acknowledge that PC members benefit from this privilege and should treat it as equally important as the rest of our duties.

In conclusion

Take what you will from my rant. I don’t want to downplay the importance of volunteers to help papers become the best they can be. But we should change our attitude with respect to expectations: nobody will kindly remind you and beg you to be on time for writing a high-quality paper, submitting an excellent proposal, or teaching an award-winning course. Why should it be different when submitting a review or engaging in a discussion?