One of the great fears for a program chair is (or ought to be) an unexpected explosion in the number of submissions. We, the co-chairs of OOPSLA 2025, certainly fretted about this, and in response put in place a Reserve Reviewer (RR) Policy (RRP). This blog post explains the policy and some outcomes. I present this not as a final, polished product but rather as a starting point for a discussion and a proposal for future chairs to grow and improve.

Problem

The problem is simple. A Web service has a simple solution to scaling: the cloud. If it ends up with more users than expected, a cloud hosting service will spin up as many additional servers as needed. In contrast, if a typical conference receives an excess of submissions, it does not have a cloud of reviewers that it can spin up as needed. (Bad jokes about cloud instances running GenAI will be sent to spam.)

Philosophy

Urban planners have a beloved slogan: you’re not “stuck in traffic”, you are traffic. Being stuck in traffic privileges your use and treats everyone else using the same roads as a nuisance. But in fact no user is more privileged than any other; traffic is the result of everyone contributing to everyone else’s woes. (The traffic analogy has obvious exceptions, like ambulances, so please don’t stretch it too much.)

In the same way, when we receive an unexpectedly large number of papers (“traffic”), no author is more privileged than any other. They are all contributing to the overload. It is therefore reasonable to consider that they might help alleviate the problem they have created. Put differently, in computer science terms, we want something that scales linearly in the number of papers, and that is roughly what the number of authors does.

How Many is Too Many?

What is an “unexpectedly large number” of submissions? We adopted the rule-of-thumb that it is reasonable to ask reviewers to review about one paper per week; much more than that seems likely to result in excessive sub-reviewing, poor review quality, and so on. For the past few years, OOPSLA has had two rounds (R1 with an October-ish deadline, R2 with a March-ish deadline). R2 receives roughly twice as many papers as R1. We therefore gave R2 roughly twice as long a review period as R1. We then used prior submission numbers as a guide, and sized the RC (Review Committee, essentially the same as a Program Committee) using the week-per-paper guideline.

This worked pretty well for R1, which received about 1.2 times as many papers as expected, many of which were revisions from 2024. This worked pretty poorly for R2, which received about 1.5 times as many papers as expected, in addition to several papers under revision. Thus we either needed reviewers to take on 150% as much work as promised, or find another solution. Fortunately, we had already put another solution in place: the RRP.

What is the RRP?

The exact text of the RRP is available in the OOPSLA 2025 Call for Papers (CFP). At its coarsest level, the RRP requires each paper to provide one or more authors to serve as reviewers if needed. We found that this kind of policy is not entirely novel: variants of this have been used, for instance, by the Association of Computational Linguistics (original policy from 2024, strengthened policy in 2025). Our version of the policy took the following form.

Each paper was expected to contribute one or more “senior” reviewers. We can view this at both a specification and implementation level. The specification we were aiming for was, “the author is a reasonable candidate to have been on the RC themselves”. We believe this is a useful way to think about whom we’d like as an RR. Put more concisely, “the community submits, and the community reviews”.

Of course, this is too subjective a specification. We therefore implemented it as follows. We defined a senior reviewer as having “completed their PhD five or more years ago”. We exempted a paper from the RRP if one of these was true:

1. The paper had no senior authors.
2. At least one senior author was already in the RC.
3. Every senior author of the paper satisfied one or more of these criteria:
   1. they had not previously published in SIGPLAN;
   2. they were chairing a SIGPLAN-sponsored conference with 150 or more submissions last year, this year, or next year;
   3. had some other exceptional circumstance that didn’t prevent writing the paper but prevented doing any reviewing.

Given our specification, we feel these criteria are fairly self-explanatory, though of course they can easily be adjusted around the edges. (Credit to Sophia Drossopoulou for helping us fine-tune the conditional logic.)

After submissions came in, we spot-checked the exempt papers. In some cases we found authors who could reasonably be considered “senior” by our specification. However, though they had long lists of publications in SIGPLAN-adjacent venues (e.g., in software engineering or security), by the letter of the above rule, they were indeed exempt. It is difficult to know how we could have included them without casting too wide a net and ending up with reviewers who have little understanding of SIGPLAN material, standards, and norms.

Unfortunately, we also ended up with some authors (presumably) not reading the rules carefully enough and declaring themselves as RRs when they ought not have been. It is therefore necessary to perform some checks on the candidate RRs before assigning papers to them.

How was the RRP Used?

In R1, we made almost no use of the RRP. It did come in handy for a very small number of papers for which we had virtually no expertise on the RC but had high expertise from trusted reviewers in the RRP. (This would be no different from finding external experts, but in our case we could do so from the RR pool itself.)

In R2 we made significant use of the RRP. We limited each reviewer to no more than 25% more than the expected number of papers. All remaining review slots were taken up by Reserve Reviewers (RRs).

There is a tricky problem of how to assign papers to RRs. The RC itself has to undergo vetting by the Steering Committee (thanks, Anders Møller). There is ample time to do this. It would be difficult to vet the RRs quickly, because they may submit a paper at the very last minute, but there is little time to lose for bidding, assigning, and reviewing. Since RR members are not vetted, we did not want to open up the full bidding process to them (where they would have access to the full papers, etc.). We were also mindful that with unvetted reviewers, there is a danger of rings of mutual peer-review support.

OOPSLA was already using the Toronto Paper-Matching System (TPMS). RRs were therefore required to register with TPMS, and we used TPMS to obtain an assignment of papers to both RC members and RRs. In situations where there was not enough expertise on the RC, we used RRs to get additional reviews.

This brings us to two controversial points.

1. TPMS is only so good (and in some cases not very good at all). Therefore, in some cases it made poor paper assignments. Frankly, there is only so much RC chairs can do in the rush to get papers assigned  — there is little time for many rounds of back-and-forth with RRs.
2. Some papers simply have no expertise at all. When given a choice, we assigned these papers to RRs, so that RC members were more likely to get papers they wanted. Thus, some RRs may have gotten very sub-optimal papers to review, but had they not, an RC member would have gotten it with the same degree of sub-optimality. Since RR members were “traffic”, we felt this was a fair trade-off. (Note that in R1, no RRs got such paper assignments.)

It is worth noting that not every low-expertise reviewer is inherently a problem! Ultimately, we would like our papers to be at least somewhat understandable beyond the narrowest sub-communities. Having a reviewer who has only a passing understanding of the material is actually helpful in providing an informed outsider’s perspective, thereby helping to raise the paper’s readability. The outsider may also spot and help prevent any nefarious behavior.

Reception

It is difficult to know how people felt about the RRP overall. What we can say is that only one member of the community reached out to explicitly complain. They felt it was an imposition on them, and they would have preferred a different process. Unfortunately, they contacted us only at the deadline of R2, so we were in no position to do anything about their comments. We do note that alternate policies that put a lot of pressure on Associate Chairs (sometimes colloquially called Associate Editors, or AEs) or individual reviewers are quite error-prone.

Failure Modes

There are some clear failure modes in the system.

An RR could clearly try to tank a paper to improve the chances of their own. In principle, so could an RC member or an AE, all of whom are also permitted to submit papers. The one difference is that an RR definitely has a paper in the system, while the others might not. Therefore, this requires vigilance, but not a whole new kind of vigilance.

An RR might lose interest in reviewing once they realize their paper is not going to be accepted. In principle, so could an RC member or AE. (We have probably all seen some signs of this, and indeed once upon a time it was either formally or informally a rule that PC members could not submit papers. Such a policy is no longer tenable, however.) Again, the main difference is that an RR definitely has a paper in the system. In addition, the others have made more of a commitment to the conference; the RR’s connection is more fleeting. Our hope is that senior community members will rise above this (for the most part).

An RR has not been vetted like RC members have been. Therefore, they may engage in nefarious behaviors. However, this is not much different than an external reviewer doing exactly the same.

An RR might simply fail to return a review. Again, this can happen with RC members and AEs, too. Perhaps surprisingly, the Chairs have more control over RRs than RC members and AEs! Because the RRP is part of the requirements of the CFP, failure to submit a review can be viewed as failing to meet the CFP requirements. The paper can thus meet the same fate as one that fails other requirements (such as anonymization, page limits, etc.): it can be desk-rejected. This was left implicit in the current CFP, but it would be good to explicate this. We note that CVPR 2025 went even further and rejected papers that would otherwise have been accepted on account of “highly irresponsible” reviewers.

In practice:

• Most of our RRs returned very reasonable reviews for the level of expertise they had. Some were high-expertise reviewers and truly engaged in the process, no different from a formal RC member. This is in keeping with the generally strong communal ethos of SIGPLAN.
• The reviewer who complained about the process also wrote a somewhat ill-tempered review, explicitly criticizing the RRP in their author-visible comments.
• Only one author pointedly refused to participate in the RR process after receiving their initial reviews (in R2), due to frustration with the quality of their reviews. Perhaps not at all coincidentally, one of their reviews was the aforementioned one with complaints. Ironically, we had put a good deal of effort into shepherding this same author’s paper in R1 over the line, but clearly they were unwilling to reciprocate.
• An extremely small number of RRs were late. In the most extreme cases of unresponsiveness, we wrote to the authors of the corresponding paper and informed them of the potential for a desk-rejection. The reviews showed up a few hours later. This is more than can be said for all RC members!

Overall, then, very, very few issues occurred in practice: enough that we can count all of them with one hand and remember all of them without writing them down.

Assessment

It is dangerous for the creators of a process to also assess it, so we refrain from doing so. We will simply say that, in our belief, it was not a failure. We certainly believe it has worked well enough that we encourage others to give it a try. Absent a root-and-branch redesign of our processes (which some of us think are worth pursuing, but SIGPLAN has too many turfs to make that feasible), it addresses some problems that would otherwise be quite sticky. We have outlined above both some issues to look out for as well as knobs to turn; we have probably missed several of each. Give it a shot, and tell us and each other how it goes!

Shriram Krishnamurthi is a professor of computer science at Brown University. He is recovering from co-chairing OOPSLA 2025; in turn, its reviewers and authors are probably recovering from him.

Sukyoung Ryu is a professor of the School of Computing at the Korea Advanced Institute of Science and Technology (KAIST). Her current research interests include mechanized language specifications and translation from C to Rust.

Disclaimer: These posts are written by individual contributors to share their thoughts on the SIGPLAN blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGPLAN or its parent organization, ACM.