Overview

Peer review is partly a lottery. A paper gets three or four reviewers, and each one notices only a handful of the issues a larger pool would raise. Swap in a different committee and the verdict can change.

We learn a checklist from thousands of past reviews and show it to every reviewer, so each committee starts from the same baseline checks. The hard part is granularity: too specific and items never recur, too broad and they fit every paper. We pick the level at which reviewers of the same paper agree most, beyond what reviewers of different papers share by chance.

1. The formulation. Checklist design as choosing a coarsening of the review claim space that removes more lottery variance than the bias it adds, selected by expected correlated agreement (ECA).
2. The pipeline. An LLM-assisted pipeline that turns a review corpus into a tree of paper-checkable checklist items at every granularity.
3. The evidence. Lower committee error in simulated review on NeurIPS 2021 and ICLR 2022, and in a 20-person human ranking study.

The problem

Submissions keep surging, and review is getting noisier and less reliable. Decisions start to look like a lottery, and a lottery rewards authors who simply try their luck, which brings in even more papers.

The NeurIPS 2021 experiment shows how noisy review already is: every paper was reviewed by two independent committees. We also broke each review into atomic claims and matched them across the two committees to see how much reviewers actually overlap.

So reviewers are noisy, and sometimes wrong. A shared checklist can fix part of this, but it has a price.

A shared checklist

A shared, LLM-verifiable checklist can reduce variance, but it can also introduce bias.

Every reviewer starts from the same checks, so scores agree more and checkable facts get checked correctly. But a checklist also steers attention. Anything it leaves out may get less weight from everyone at once, which moves the committee’s average away from the target: the score the whole reviewer population would give, μ_i(∅).

Method

A checklist is a coarsening of the claims reviewers can make. The method picks how coarse it should be with a proxy that past reviews can measure.

Choosing k by agreement

Past reviews cannot tell us how reviewers would have scored with a checklist they never saw, so we cannot measure error directly. We measure agreement instead: how often two reviewers of the same paper raise the same item (A_same), minus how often reviewers of two different papers do (A_diff). Broad items inflate both terms, narrow items shrink both, and the gap peaks in between. We prove that, under stated conditions, the ECA-maximizing checklist approximates the one with the lowest error.

Building the checklist tree from historical reviews

The pipeline reads a historical review corpus and grows one tree of checklists. Cutting the tree near the root gives a few broad items; cutting near the leaves gives thousands of narrow ones. Each cut is a candidate value of k.

1

A review sentence

NeurIPS 2021

“Finally, the experimental evaluation omits some important baselines such as DISCERN, which was also evaluated on Atari.”

2

Split, filter LLM

15,118 claims from 298 papers

atomic claim the paper can settleThe paper’s experimental evaluation does not include a comparison to the DISCERN baseline.

3

Strip names

so papers can match

The paper’s experimental evaluation does not include a comparison to the [METHOD] baseline.

4

Embed, merge LLM

only exact repeats merge

this claim…does not include a comparison to the [METHOD] baseline.

nearby, another paper…does not include comparisons to non-trivial baseline algorithms…

5

Cluster, name LLM

one tree, many k

k = 100“Are baselines and SOTA comparisons adequate?”289 claims

k = 500“Does the paper compare against experimental baselines?”67 claims

k = 6,000“Missing important experimental baselines”12 claims

Show the paper’s original pipeline figure

Results

Do the learned checklists reduce review error, in simulation and in a human study?

A stable target to measure error against

To evaluate error we need the target μ_i(∅), but we cannot aggregate infinitely many human reviewers. Instead we aggregate LLM-simulated reviewers with no checklist until the aggregated score converges. For each of 30 held-out papers per venue (10 strong, 10 borderline, 10 weak), we draw one unaided reviewer for each content branch of the tree at 50 items: 49 on NeurIPS and 47 on ICLR. Each reviews its branch and scores the paper, and the average over all of them is the target.

A checklist cuts error

Three reviewers: variance and bias²

No checklist

0.27

Learned, k = 100

0.12

Official list

0.16

variancebias²

Show the paper’s original variance, bias and MSE figure

With the 100-item checklist, a single reviewer’s error on NeurIPS drops from 0.82 to 0.21, and a committee of three has 58% lower error; a committee of four halves its error (0.21 to 0.10). The official NeurIPS 2021 checklist lowers variance just as much but adds more bias, leaving it about 0.04 worse at every committee size. On ICLR, a single reviewer’s error drops from 0.84 to 0.34.

The k with the highest ECA has the lowest error

ECA is computed from past reviews only, so it needs no checklist to have been used. We check it against the error of simulated four-reviewer committees, one point per checklist size. On both venues the size with the highest ECA, k = 100, has the lowest error, and the size with the lowest ECA, k = 2,000, has the highest (0.104 against 0.180 on NeurIPS, 0.205 against 0.287 on ICLR).

People also rank better with the checklist

In a human study, 20 participants each ranked two sets of three NeurIPS 2021 papers on the same topic, one with the k = 100 checklist and one without (topic and condition counterbalanced across participants). The checklist was filled in by an LLM beforehand. Error is the share of pairs ranked in the wrong NeurIPS order. It fell from 0.48 to 0.22 for one participant (−54%), and from 0.32 to 0.11 for a committee of three (−66%).

Takeaways

1. 1
   Formalize the checklist objective as a variance-bias tradeoff. Review error is variance plus bias², and a checklist moves both.
2. 2
   Propose ECA as a proxy objective for review error. On both venues, k = 100 has the highest ECA and the lowest error.
3. 3
   Formulate checklist design as coarsening the review claim space. k items, from broad to narrow: the single knob is the granularity.

What this does not claim

• Only paper-checkable claims are on the list. Novelty and field impact stay off it, and that omission is the bias.
• Simulated review uses LLM reviewers, each covering a narrow slice of concerns, on 30 papers per venue. The target is that simulated population, not a human meta-review.
• The human study is a ranking task with a checklist an LLM filled in first; it does not test people writing reviews from a blank checklist.
• Acceptance decisions are not measured. We do not claim a checklist should replace reviewers or change what a venue accepts.

Team

This is research from the Epistemic Decisions Lab at Northwestern University by:

• Xuanyou Liu
• Ziyang Guo
• Jason Hartline
• Jessica Hullman

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review Under review at ICLR 2027 · Oral and Best Paper Honorable Mention, NeurIPS 2026 Workshop on AI-Native Academia · xuanyouliu.com

Publication

This work is under review at ICLR 2027. The preprint is available on arXiv or as a direct PDF. A workshop version, Learning Automated Checklists to Reduce Lottery Effects in Scientific Peer Review, was accepted as an oral presentation at the NeurIPS 2026 Workshop on AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI, where it received a Best Paper Honorable Mention.

@misc{liu2026checklists,
  title={Learning Checklists to Reduce Lottery Effects in Scientific Peer Review},
  author={Liu, Xuanyou and Guo, Ziyang and Hartline, Jason and Hullman, Jessica},
  year={2026},
  eprint={2610.28666},
  archivePrefix={arXiv},
  primaryClass={cs.DL},
  note={Under review. Workshop version: Best Paper Honorable Mention, NeurIPS 2026 Workshop on AI-Native Academia}
}