To check whether a review comment misstates the paper, PremiseCheck lets an LLM rewrite the comment as explicit premises without reading the paper and lets a cheap decision model verify each premise against the paper. When each comment is checked on its own, it reaches accuracy comparable to holistic GPT-5.5 on all three fields of our ReviewFact benchmark at about half its cost, and at a quarter of its cost on NLP reviews.
Abstract
Can review comments be fact-checked without an LLM reading the paper each time?
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says.
We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy.
A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions.
PremiseCheck is a framework that combines the two.
An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper.
When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.
The LLM never reads the paper
It only rewrites the comment, which averages 24 tokens on the NLP reviews, as one to six self-contained premises, explicit and implicit.
A decision model reads it instead
It returns the probability that a statement holds and generates no text. All its calls together cost $0.0015 per comment on the NLP reviews.
Each flag names a premise
A comment gets the score of its weakest premise, so a flagged comment comes with the premise that the decision model finds false.
The idea
Three ways to check a review comment
Figure 1. Three ways to check a review comment, shown on a real comment from ReviewFact. AUROC, cost and time are measured on the NLP reviews, with cost relative to the LLM-only row. The LLM is GPT-5.5 and the decision model is Jev.
Decision model only. A decision model is cheap and fast, but on whole comments it is near chance on the NLP reviews (AUROC 49.9). A comment mixes several assertions, many of them implicit, while a decision model is reliable only on explicit, self-contained claims.
LLM only. A frontier LLM judging each comment holistically is accurate (AUROC 63.2) but must reread the full paper for every comment, a cost that multiplies across tens of thousands of reviews.
PremiseCheck. The LLM turns the comment into explicit premises from the comment alone, and the decision model checks each one against the paper. In this example the premise that the paper tests reproducibility only on the Gopher model gets 0.03, and the comment is caught. On the NLP reviews this setup reaches AUROC 63.1 at 25% of the cost and 2.8 s per comment.
Method
How PremiseCheck works
PremiseCheck scores a comment on a paper in four steps. Only the first step uses an LLM, and it never sees the paper. Any LLM and any decision model can fill the two roles.
Step 1 LLM
Decompose
The LLM reconstructs the comment as an argument with one to six premises. Each is one self-contained declarative sentence that names the method, table or section it refers to, including what a question presupposes about the paper.
Step 2 Decision model
Route
From the premise alone, the decision model estimates whether it can be decided from the paper (“the paper does not report results on dataset X”) or relies on general knowledge (“method Y is a standard baseline”).
Step 3 Decision model
Verify
The decision model reads the paper with the premise and answers the question that fits its route: “true according to the paper” for paper premises, “true” for knowledge premises.
Step 4 min
Aggregate
A comment that rests on a false premise contains a factual error, so the comment score is the lowest premise score. A low score flags the comment for a person to check.
Scroll sideways to see the full figure →
Figure 2. PremiseCheck on a ReviewFact sample that experts labeled as a factual error, with premises shortened for display. Only the verifier reads the paper, and the lowest premise score, shown in red, is the comment score.
Why it is cheap. The LLM call is short because it excludes the paper. A decision model generates no text, so its cost grows only with the length of its input, and asking several questions about the same input adds almost no cost. In holistic judgment, by contrast, the LLM reads the paper and reasons about it for every comment, so its cost grows with the length of the paper at LLM prices.
Benchmark
ReviewFact: factual errors in human and AI review comments
Given a paper and one comment from a review of it, a system scores how likely the comment says something the paper does not support. ReviewFact adds no new comments or papers: it maps the expert judgments of ReviewCritique (NLP), ReviewScore (machine learning) and PeerReview Bench (natural sciences) to one definition of factual error, and it measures cost and latency alongside AUROC.
What counts as a factual error. The comment says something about the paper that the paper does not support. Following ReviewCritique, it either
Misunderstandingmisreads the paper,
Neglectclaims that the paper lacks something it contains,
Unstated statementasserts something with no basis in the paper, or
Inaccurate summarymisstates its content.
Errors of judgment, such as whether a concern is important, are outside the task.
Figure 3. ReviewFact by part and reviewer type: 9,267 comments on 217 papers. Left, the share of comments with a factual error, next to the counts of errors and comments. Right, the types of these errors, where Other is Unstated statement or Inaccurate summary. Errors are rare, so a system spends most of its calls on correct comments and its cost matters.
Label check. On the natural-science reviews, GPT-5.5 decides from the expert's rationale whether an error is factual. Two authors independently repeated this decision on the 178 comments with a rationale: they agree with each other at Cohen's κ = 0.70, and GPT-5.5 agrees with them at 0.76 and 0.77. With the human labels instead, the difference between PremiseCheck and holistic GPT-5.5 ranges from −3.7 to +2.7 and is never significant.
Results
Accuracy comparable to holistic GPT-5.5 at a fraction of the cost
With each comment checked on its own, PremiseCheck with a medium-reasoning decomposer is within 0.3 points of holistic GPT-5.5 or above it on all three parts of ReviewFact, at about half the cost. It is non-inferior on the NLP reviews and not significantly different on the other two parts.
9,267
labeled review comments on 217 papers across three fields
46.6–51.2%
of the reference cost with a medium-reasoning decomposer on the three parts
24.6%
of the reference cost on NLP reviews with a no-reasoning decomposer (AUROC 63.1, non-inferior)
45%
of the reference time on NLP reviews with a no-reasoning decomposer (37% and 22% on the other parts)
Table 1. Results on ReviewFact with each comment checked on its own
AUROC × 100 with the paired difference to the reference, computed before rounding, as a subscript. † marks non-inferiority: the lower bound of the 95% interval for the difference is above −3. Cost and time are % of the reference, holistic GPT-5.5 at medium reasoning. Reasoning refers to the GPT-5.5 or GPT-5.4-mini call, which for PremiseCheck is the decomposer. Bold marks the best AUROC per part.
System
Reasoning
NLP
Machine learning
Natural science
AUROC
Cost %
Time %
AUROC
Cost %
Time %
AUROC
Cost %
Time %
Holistic judgment
GPT-5.5 (ref.)
medium
63.2
100.0
100.0
66.9
100.0
100.0
67.8
100.0
100.0
GPT-5.5
low
61.2−2.1
76.1
67.6
64.3−2.6
120.5
67.9
67.4−0.4
75.5
46.0
GPT-5.5
none
49.3−14.0
49.9
28.4
66.5−0.4
95.7
26.1
63.7−4.1
54.2
13.7
GPT-5.4-mini
low
53.7−9.5
9.8
31.1
58.7−8.2
13.5
36.6
61.4−6.4
13.8
34.4
Jev
–
49.9−13.4
2.1
3.6
61.2−5.7
1.6
3.6
58.7−9.1
1.6
1.9
Cascade
Jev → GPT-5.5
medium
59.1−4.2
74.0
–
65.7−1.1
76.4
–
66.7−1.1
69.2
–
PremiseCheck (decomposer: GPT-5.5, verifier: Jev)
PremiseCheck
medium
63.0†−0.3
51.2
81.6
68.7+1.8
46.6
95.7
67.5−0.3
47.4
60.6
PremiseCheck
low
64.1†+0.8
34.3
60.1
63.9−3.0
29.2
52.9
65.7−2.1
26.2
28.1
PremiseCheck
none
63.1†−0.2
24.6
45.0
63.7−3.2
19.5
37.5
66.0−1.7
22.3
21.9
Scroll sideways to see all three parts →
PremiseCheck (ours)Baselines; cheaper variants in greyOn machine learning and natural science, no PremiseCheck difference to the reference is significant.
Cost versus accuracy on the NLP reviews
Figure 4. Cost and AUROC on the NLP reviews, (a) per comment and (b) batched per review, where PremiseCheck uses 16 phrasings. Labels give the reasoning of the LLM; the line and band mark the per-comment reference and ±3.
(a) Per comment. At medium, low and no reasoning, PremiseCheck stays inside the ±3 band around holistic GPT-5.5, while its cost falls from 51.2% to 24.6% of the reference.
Cheaper per-comment baselines lose accuracy. Holistic Jev and GPT-5.4-mini fall 5.7 to 13.4 points below the reference. Holistic GPT-5.5 without reasoning falls 14.0 on the NLP reviews at twice the cost of PremiseCheck, and the cascade costs about three quarters of the reference yet stays below PremiseCheck with the medium-reasoning decomposer on every part.
(b) Batched per review. When a review is finished, all its comments can be checked in one call; Finding 3 gives the numbers.
Findings
What makes PremiseCheck work, and how to use it
Decomposition moves the work that needs reasoning from the LLM to the decision model. The LLM can then run without reasoning, and decision models from other providers, including an open-weight one, can check the premises.
Finding 1
Decomposition makes decision models useful
Figure 5. AUROC on the NLP reviews of each model as a holistic judge and inside PremiseCheck. Cost is relative to holistic GPT-5.5 in (a) and to the same LLM in (b), where * marks API cost. The band marks holistic GPT-5.5 ±3.
Every decision model gains from decomposition. Jev, Kev-27B, Microsoft Decision-1 and Perplexity Decider score 50 to 57 as holistic judges but come within one point of holistic GPT-5.5 as verifiers, although the intervals of Kev-27B and Perplexity Decider fall just short of non-inferiority.
A cheap verifier is enough. On the same premises, GPT-5.5 as the verifier is no more accurate than Jev but makes the pipeline cost 3.1 times as much as holistic judgment.
Much cheaper LLMs can decompose. With Kev-27B as the verifier, GPT-5.6 Sol, GPT-5.6 Luna and Nemotron Super 3 are non-inferior to holistic GPT-5.5 at 6% to 17% of the cost of their own holistic judgment, while weaker decomposers lose 2 to 6 points.
Finding 2
An open-weight decision model works as well
Figure 6. AUROC of PremiseCheck minus holistic GPT-5.5 on the NLP reviews, with 95% intervals; the dashed line marks the non-inferiority margin of −3.
Kev-27B instead of Jev. With the open-weight Kev-27B as router and verifier, PremiseCheck reaches the same AUROC as with Jev. With the no-reasoning decomposer it is non-inferior at an API cost of 17.8% of the reference, not counting the GPU time of Kev-27B; with the medium-reasoning decomposer its interval falls just short of non-inferiority.
Not a proxy for comment length. Longer comments are more often factual errors, but within groups of similar length and after combining each score with length, PremiseCheck stays within 3 points of holistic GPT-5.5 without a significant difference, though none of these comparisons reaches non-inferiority. Adding PremiseCheck to length raises AUROC significantly, by 3.3.
Finding 3
Batching helps both approaches, and fusing them gives the best result
When a review is finished, as for an area chair or an author, all its comments can be checked in one call. We batch both approaches per review and fuse them by averaging the percentile ranks of their scores.
Batched holistic GPT-5.5 without reasoning and batched PremiseCheck do not differ significantly, so holistic GPT-5.5 is the cheaper choice in this setting, while PremiseCheck also returns the premises. The two scores have a rank correlation of only 0.56, and their fusion is significantly above batched holistic GPT-5.5.
67.8
Batched holistic GPT-5.5, no reasoning, at 7.7% of the reference cost
69.0
Batched PremiseCheck, no reasoning, 16 phrasings, at 20% of the reference cost
71.5
Fusion of the two at 28% of the reference cost
AUROC on the NLP reviews.
Finding 4
Flagging comments for a person to check
Figure 7. Flagging factual errors on the NLP reviews, where 4.8% of comments contain one. Hit rate is the share of papers with an error that one of the three flags hits. PremiseCheck uses 16 phrasings, and ties are broken at random.
Flags hit errors two to three times as often as random. This holds for PremiseCheck and batched PremiseCheck among the 5% most suspicious comments, whereas holistic GPT-5.5, per comment or batched, does little better than random despite a similar AUROC.
An area chair sees more errors. Reading the three most suspicious comments of each paper finds an error for 40% of the papers that contain one with batched PremiseCheck, compared with 29% with batched holistic GPT-5.5.
A flag points to the premise that the paper refutes, as in Figure 8.
Figure 8. A flagged comment from the machine learning reviews with the authors' OpenReview response, quoted verbatim. Jev gives the premise that the paper has no summary of its takeaways a probability of 0.04, and the authors reply that each experiment section ends with one.
Design choices
What each part of the pipeline contributes
Routing is needed, but the exact route is not. Kev-27B agrees with Jev on only 65% of the routes yet gives the same AUROC, whereas skipping routing lowers AUROC from 61.8 to 58.8 on a common set of machine learning premises, because knowledge premises then receive low probabilities.
The weakest premise decides. Taking the mean over premises instead of the minimum lowers AUROC to 59.4.
Contradiction questions add accuracy at no cost. Asking 16 phrasings of the verification question in one decision-model call, half of them whether the paper contradicts the premise, raises AUROC on the NLP reviews from 63.1 to 65.7, a significant gain over the original question.
The decomposer needs no reasoning. Without reasoning, per-comment holistic judgment falls to chance level on the NLP reviews, while PremiseCheck keeps its accuracy because the LLM only rewrites the comment.
Limitations
What PremiseCheck does not solve
The task is hard. Each error is something an expert reviewer overlooked or misread, and the evidence against it may lie anywhere in a long paper. On the NLP reviews, per-comment AUROC is 63.2 for holistic GPT-5.5, 63.0 to 64.1 for PremiseCheck in Table 1, and 65.7 with 16 verification phrasings.
The machine learning and natural-science parts are small. They contain 46 and 65 factual errors, and with the medium-reasoning decomposer the 95% intervals for the difference to holistic GPT-5.5 are [−9.4, +11.7] and [−8.9, +7.6]. On these parts the difference is not significant; non-inferiority is shown on the NLP reviews only.
Facts, not judgment. PremiseCheck treats a comment as wrong when one premise is false, which does not hold for errors of judgment, such as whether a concern matters or how novel the work is. On labels that also count such errors, it falls behind holistic GPT-5.5. Detecting errors of judgment is left for future work.
Where it fails. In a sample of failures, false alarms on the NLP reviews come mostly from decomposition (58%), where a premise adds to, redirects or blurs what the reviewer said, and on the natural-science reviews mostly from verification (45%).
Model availability. The main decision model, Jev, is served through an API. With the open-weight Kev-27B, PremiseCheck reaches the same AUROC on the NLP reviews; further open decision models are untested.
Intended use. A verifier that wrongly flags a correct comment could unfairly discredit a reviewer, and one that misses an error could lend false authority to a wrong comment. PremiseCheck is meant to direct human attention to comments that may misstate the paper, not to replace the judgment of reviewers, area chairs or authors.
Release. The NLP and natural-science parts of ReviewFact are released under the licenses of ReviewCritique and PeerReview Bench, which allow evaluation but not training. ReviewScore states no license, so for the machine learning part we release a script that rebuilds it from the original data. The code is released under the MIT license.
BibTeX
Citation
@article{song2026premisecheck,
title = {{PremiseCheck}: Efficient Fact-Checking of Human and {AI} Peer Reviews},
author = {Song, Dingjie and Yan, Zhiling and Zhang, Ziqian and Zhang, Yuxuan and Sun, Lichao},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}
