Northwestern University Epistemic
Decisions Lab

NeurIPS 2026 AI-Native Academia workshop

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman

With surging paper submissions, noisy and wrong peer review leads to a “Death Spiral”

Noisy

of papers got different decisions from two independent committees

False criticism

18%

of reviewers’ claims can be refuted by the paper itself

Beygelzimer et al. 2023

Death Spiral

More & More Papers
AI-fueled surge Overload / Noise
Review fatigue Random
Lottery Spam Paid Off
Even more papers Review
death spiral

Liu et al. 2023

Venues already use expert-written checklists to reduce review noise

1

Examples

many venues now have one
Many venues now use an official checklist

NeurIPS Paper Checklist · ACL Rolling Review Responsible NLP Checklist and reviewer checklist · AAAI Reproducibility Checklist

2

Who writes it

experts
A list of points to watch for

Experts decide what reviewers should look at, giving everyone a shared basis for judging a paper.

3

Why it might help

a shared starting point
Every reviewer runs the same baseline checks

So scores depend less on which reviewers a paper happens to get.

A shared checklist helps with variance

but it introduces bias

A shared checklist reduces variance but introduces bias

score reviewer lottery variance target μi(∅) R1 score 3 “DISCERN missing” R2 score 5 “No ablation” R3 score 6 “Weak baseline” Draw 1 mean R1 score 5 “No ablation” R2 score 7 “Gap in Theorem 2” R3 score 8 “Strong results” Draw 1 mean Draw 2 mean Included in the checklist ✓ DISCERN used? ✓ Fusion ablated? ✓ Split reported? lower variance checklist mean μi(g) bias

Error of the committee mean

MSE=Variance+Bias2

A checklist helps when

Var(∅) − Var(g)variance removed

>

(μ(g) − μ(∅))2bias² added

g: with checklist · ∅: no checklist

A checklist is not always good

We need to formalize checklist generation to find a good one, responsibly

Learn the checklist from historical reviews

1

Not written by hand

learn it from data
Learn a checklist from historical data

Instead of experts writing the list by hand, let past reviews show what reviewers raise.

2

Our view

a checklist is a coarsening
A checklist is a coarsening of historical reviews

Specific past claims are grouped into broader, shared items. The coarsening level sets how broad they are.

3

Learning the checklist

find the right level
Find the level that minimizes review MSE

The right coarsening helps reviewers reach the lowest error.

Ideally, run a human experiment to find the optimal coarsening level

score all possible reviewers target μi(∅) Checklist A ✓ DISCERN used? ✓ Fusion ablated? ✓ Split reported? variance checklist mean μi(g) bias Checklist B ✓ Is the idea novel? ✓ Is the impact clear? ✓ Are claims supported? variance checklist mean μi(g) bias Checklist C ✓ Baselines adequate? ✓ Ablations reported? ✓ Is the writing clear? variance checklist mean μi(g) bias

Human experiment

All possible reviewers give the target μi(∅)

Run reviews with different coarsening levels

MSE=Variance+Bias2

A 0.46 B 0.34 C 0.29

Choose the one with the smallest MSE

But this is impossible with human reviewers

So we use a proxy: Expected Correlated Agreement (ECA)

Compare reviewer agreement under different checklists. In theory, it finds the optimal coarsening level.

Pick the k with the biggest agreement gap (ECA), a proxy for error

Past reviewers never saw a checklist, so we cannot measure its error. We measure agreement.

0%10%20%051015 501002005001,0002,0006,000 k = number of checklist items (log scale) +9.8+10.2+9.0+7.3+5.9+4.2+1.8 +10.2 ECA k = 100 Asame: same paper Adiff: different papers gap = ECA
Asame

Two reviewers of the same paper raise the same item.

Adiff

Two reviewers of different papers raise the same item.

ECA = Asame − Adiff

The gap peaks in the middle.

The peak

k = 100 on both venues: +10.2 NeurIPS, +12.9 ICLR.

Our pipeline

Build a checklist tree from historical reviews, then pick the level with ECA

Build a checklist tree from historical reviews

1

A review sentence

NeurIPS 2021
“FastNet is never compared with XGBoost, which is the standard strong baseline for tabular data.”
2

Split, filter LLM

15,118 claims kept
atomic claim the paper can settleThe paper does not compare FastNet with XGBoost.
3

Strip names

so papers can match
The paper does not compare [METHOD] with XGBoost.
4

Embed, merge LLM

only exact repeats merge
this claim…does not compare [METHOD] with XGBoost.
nearby, another paper…Random Forest is not included as a baseline for [METHOD].
5

Cluster, name LLM

one tree, many k
k = 100“Is the method compared against baselines?”
k = 200“Are strong tabular-data baselines included?”
k = 6,000“No comparison to XGBoost”

Evaluation

Do the learned checklists reduce review error in simulation and human studies?

Simulate a stable ground truth to evaluate the error

30 held-out papers per venue. Every reviewer is an LLM agent.

−10+1 committee mean − target (points) 14163249 unaided reviewers in the committee, N target 80% of random committees ±1.11 ±0.53 ±0.16

Review error of the committee mean

MSEi(g) = E[(Yi(g) − μi(∅))2]

= Var(Yi(g)) + (μi(g) − μi(∅))2

  • To evaluate error, we need the target μi(∅).
  • We cannot aggregate infinitely many human reviewers.
  • We aggregate LLM-simulated reviewers with no checklist, until the aggregated score converges.

A three-reviewer committee has 58% lower error with a checklist

Simulated review on 30 held-out NeurIPS 2021 papers.

0.00.20.40.60.8 13481216 committee size N (reviewers) error of the committee mean (MSE) 0.82 No checklist (unbiased) 0.21 Learned checklist, k = 100 −58% at N = 3 Official NeurIPS 2021 list N = 10 unaided catches up N = 7

The k with the highest ECA has the lowest error

Each point is one checklist size k. Error is from simulated review with N = 4 reviewers.

NeurIPS 2021r = −0.84 (n = 5 sizes)

0.100.140.1846810 ECA (higher is better) Error (lower is better) k = 50k = 100k = 500k = 1,000k = 2,000

ICLR 2022r = −0.81 (n = 5 sizes)

0.200.240.2810111213 ECA (higher is better) Error (lower is better) k = 50k = 100k = 500k = 1,000k = 2,000

People also rank better with the checklist

20 participants. Checklists filled in by an LLM.

0.0 0.2 0.4 1 2 3 4 5 committee size N (participants) ranking error (MSE) 0.48 0.32 No checklist 0.22 0.11 Checklist, k = 100 −54% error at N = 1 −66% error at N = 3

Task

Rank 3 papers on the same topic with/without a checklist.

Example: pruningNeurIPS score

Sparse Training via Boosting…7.0

CHIP: CHannel Independence…6.0

Efficient Neural Network…5.0

Error: pairs ranked against the NeurIPS order.

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

1

Formalize the checklist objective in terms of the variance-bias tradeoff

variance + bias²a checklist moves both
2

Propose ECA as a proxy objective for review error

k = 100highest ECA, lowest error
3

Formulate checklist design as coarsening the review claim space

k itemsfrom broad to narrow
First page of the paper: Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Read the paperxuanyouliu.com/projects/
review-checklists

  • Xuanyou LiuXuanyou Liu
  • Ziyang GuoZiyang GuoOn the job market
  • Jason HartlineJason Hartline
  • Jessica HullmanJessica Hullman
Epistemic Decisions Lab · Northwestern University NeurIPS 2026 AI-Native Academia workshop

Backup · agreement at eight cuts

ECA peaks at 100 items on both venues

Items k NeurIPS 2021 ICLR 2022
Asame Adiff ECA Asame Adiff ECA
5022.112.4+9.838.927.2+11.7
10016.26.1+10.228.815.9+12.9
20012.33.2+9.021.79.2+12.6
5008.71.4+7.315.84.2+11.6
6008.21.1+7.114.93.5+11.5
1,0006.60.7+5.913.12.3+10.8
2,0004.50.4+4.210.91.2+9.8
6,0002.00.1+1.88.10.4+7.7

All values are percents. The highlighted row is the checklist reviewers see.

Backup · limits

What this does not claim

What is on the list Paper-checkable claims

Novelty and field impact stay off the list. That omission is the bias.

Simulated review 30 papers per venue

LLM reviewers, each covering a narrow slice. The target is that population, not a human meta-review.

Human study Ranking, prefilled

20 people ranked papers. The checklist was filled in by an LLM first.

Not measured Acceptance decisions

We do not claim the checklist should replace reviewers, or that it changes what a venue accepts.

Backup · pipeline as in the paper

Eight steps, three stages

Three stages. Reviews are split into atomic claims and filtered to facts the paper can settle. Claims are stripped of paper-specific names, embedded, and merged. Clustering grows one tree, from broad questions at 100 items to narrow ones at 6,000.

Backup · variance, bias and MSE on NeurIPS 2021

Variance falls, bias stays small

NeurIPS 2021 committee error from 1 to 16 reviewers. The learned checklist cuts variance sharply, adds a small bias, and stays below unaided review until about ten reviewers. The official checklist cuts variance too, but adds more bias.