Stable but Input-Dependent: Detecting Generic LLM Critiques by Perturbing Both the Prompt and the Paper

Language models asked to review a scientific paper tend to write feedback that sounds right and helps no one. Typical examples are "the experiments could be more thorough" or "consider discussing limitations." Existing fixes train the model on better feedback or wrap it in slow multi-agent pipelines. This note proposes a test-time diagnostic instead. A useful comment should survive rewording of the instructions, and it should disappear when the part of the paper it complains about is changed. A generic comment fails the second test: it survives almost any edit to the paper. A hallucinated comment fails the first: it appears under one phrasing of the prompt and vanishes under the next. Scoring each comment on both axes should separate valid, specific critiques from boilerplate and from fabrications. It needs no labels and no training, and it works on any model.

The opportunity comes from putting two lines of work side by side. Prompt-sensitivity research shows that when a model's answer flips under harmless rewordings or reordered examples, the answer is probably wrong (On the Relation between Sensitivity and Accuracy in In-Context Learning). That work was done only for classification labels. Feedback research shows that generic comments are the dominant failure of LLM reviewers (GoodPoint; SWIF²T). Nobody has checked the obvious catch. Generic boilerplate is probably highly stable under prompt perturbation, because it does not depend on anything specific. Prompt stability alone should therefore reward exactly the wrong comments. The second axis, dependence on the document, is the missing ingredient.

Core Claim

We claim that the validity of an individual critique comment, as judged by the paper's authors, is predicted better by the product of two quantities than by either quantity alone, and better than single-pass confidence. The first quantity is the comment's stability across perturbed prompts. The second is its sensitivity to targeted counterfactual edits of the document. Both are defined below.

Concretely, a model reviews document $x$ under $K$ prompt variants $p_1,\dots,p_K$. These are instruction paraphrases plus reorderings of the in-context demonstrations, meaning the worked examples placed in the prompt. Its comments are split into atomic claims and clustered by meaning, so the same claim can be recognized across runs. For a claim $c$ that points to a passage in $x$, we build $M$ counterfactual documents $\tilde{x}_1,\dots,\tilde{x}_M$. In each one, that passage is either replaced by the corresponding passage from a different paper or repaired to address the claim. The score is then:

$$ \mathrm{Score}(c) = \underbrace{\frac{1}{K}\sum_{k=1}^{K}\mathbf{1}[c \in G(x,p_k)]}_{\text{prompt stability } S(c)} \;\times\; \underbrace{\Big(1-\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}[c \in G(\tilde{x}_m,p_1)]\Big)}_{\text{document dependence } D(c)} $$

Here $G(x,p)$ is the set of claim clusters the model produces for document $x$ under prompt $p$, and $\mathbf{1}[\cdot]$ is 1 when the claim is present and 0 otherwise.

The predicted pattern has three parts. Valid, specific critiques should have high $S$ and high $D$. Generic comments should have high $S$ and low $D$. Hallucinations, including the documented failure of flagging content as "missing" when it is actually present (GoodPoint), should have low $S$. The "missing content" failure also gets a direct counterfactual: if the paper already contains the supposedly missing material, adding it again should not remove the claim. A near-zero $D$ therefore flags the claim.

Research Program

The work has three stages, and each can fail on its own terms.

The first stage is descriptive. We measure how feedback changes under prompt perturbation, which prior work has not done. LLMCrit showed that splitting guidelines into small groups of 3–5 criteria, plus demonstrations, raises constructiveness by 25.8%. However, it used one fixed wording and one fixed demonstration order. ProSA found that going from zero to one demonstration gives the largest drop in sensitivity, while SENSEL found that demonstration order alone still carries a useful uncertainty signal. We reuse LLMCrit's criteria and demonstrations, and ProSA's four template families (simple input, emotional support, role player, output requirement). With these we measure how often the set of claims changes across variants, and whether the changes concentrate in the claims authors rejected.

The second stage builds the counterfactual edits. For each claim, we find the passage it targets. Localization follows the claim-level decomposition of RAGChecker and the span-level approach of Localizing and Mitigating Errors in Long-form QA. We then make two kinds of edit. A swap replaces the passage with the same section, for example "Experimental Setup," taken from another paper in the corpus. A repair uses a separate model to rewrite the passage so that it addresses the claim. Claims that cannot be localized receive only swap edits of the whole section. We report how often that happens, because generic comments may be exactly the ones that resist localization.

The third stage puts the score to use. It serves as a selective filter, dropping low-scoring comments, and as a best-of-$K$ reranker that assembles a final review from high-scoring claims. Both run on base models and on fine-tuned critics.

The in-context-learning mechanism literature constrains this design more than it supplies it. In-Context Learning Creates Task Vectors shows that demonstrations are compressed into a compact internal activation. One Task Vector is not Enough and Understanding Task Vectors show that such single vectors fail on open-ended, multi-step generation. The latter reports a drop from 100% to 55% on a bijection task. Steering a critic with an injected "rigor vector" is therefore a poor bet. We treat demonstrations as a perturbation source and anchor, not as a steering handle.

Baselines

The naive reference point is to keep every comment a zero-shot critic produces, unfiltered and unranked, using base Qwen3-8B and Llama 3-70B-Instruct. The second model is included because ProSA found it unusually stable. Its comment-level validity rate on the GOODPOINT-ICLR held-out ICLR 2026 split, scored against author responses, sets the floor.

The strongest existing method is GOODPOINT fine-tuning (GoodPoint), which trains Qwen3-8B with supervised fine-tuning and then preference optimization on author rebuttals and on synthetic vague comments. In its 13-author double-blind study, validity rose from 41.5% for the base model to 58.1% for GOODPOINT-DPO. It also reports an 83.7% improvement in predicted feedback success. We compare against it in two ways. First, we rerun the GoodPoint recipe on the same training split, or use its released checkpoints if available. Our test-time filter on the base model is then compared with GoodPoint-DPO at matched output volume, meaning the same number of comments per paper. Second, we apply our filter on top of GoodPoint-DPO, since the two are complementary rather than exclusive. Our method beats this baseline if base-model-plus-filter matches GoodPoint-DPO's author-judged validity. It also counts as a win if GoodPoint-plus-filter raises validity further at a coverage of at least 70% of comments. We found no reported comment-level validity-ranking AUC on Lacuna for GOODPOINT-ICLR, so we will establish that number ourselves.

For uncertainty-based ranking, the strongest competitors are two single-pass signals: mean token probability and entropy over each claim's tokens. We also run SENSEL ported to generation, which amounts to our $S$ term alone. SENSEL reports +4.1 area under the F1–coverage curve over MaxProb and entropy on GPT-J-6B classification, and +3.0 from order permutation alone. All ranking methods get the same $K$ and the same variants.

Three ablations isolate the key ingredient. $S$-only tests whether document dependence adds anything. $D$-only tests whether prompt perturbation adds anything. A cheap proxy for $D$ tests whether the counterfactual edits are needed at all. The proxy is the maximum embedding similarity between a comment and comments the same model wrote for other papers. For the paragraph-level setting, we also compare against the full SWIF²T pipeline. It reports a helpfulness dominance score of 171.75, against 92.50 for GPT-4 and 121.25 for human reviews, at 7 minutes 11 seconds per paragraph. We report our wall-clock cost next to it.

Why not just…?

Why not just fine-tune on author responses, as GoodPoint does? This is the most serious competitor, and we treat it as a baseline rather than dismissing it. Two findings in the file suggest fine-tuning leaves room. GoodPoint itself reports that its trained model still critiques papers for content that is already present. Dynamic Evaluation with Cognitive Reasoning shows that behaviour learned through fine-tuning can drop by as much as 35% under minor system-prompt rewording. Fine-tuning also requires rebuttal data, which exists for ICLR but not for code review or student essays. The layered experiment settles the question. If filtering adds nothing on top of GoodPoint-DPO, the diagnostic is redundant wherever training data exists, and we will say so.

Why not just flag generic comments by their similarity to comments written for other papers? This is a real threat, because it is a one-pass, cheap approximation of document dependence. It will likely catch pure boilerplate such as "add more baselines." It cannot catch comments that are tailored in wording but not grounded in the paper. It also cannot catch the "missing content" error, where the comment is specific but false. Only an edit to the document reveals those. The embedding-proxy ablation settles it. If the proxy matches $D$ on author-validity AUC, we adopt it and drop the counterfactual edits. We would then report that result as a much cheaper method.

Why not just ask a strong LLM judge whether each comment is specific and actionable? LLMCrit's layered judge reached up to 96% agreement with experts, so this is plausible. However, a judge reading only the comment and the paper shares the reviewer's own blind spots. Rethinking Hallucinations found that standard detectors track consistency rather than correctness. CritiqueLLM documents judges favouring outputs from their own model family. We include a same-model and a cross-family judge as baselines, and we measure agreement with author responses rather than with other models.

Why not just use single-pass token confidence? SENSEL beat it on classification. ProSA links sensitivity to flat token distributions, which suggests confidence and stability overlap. However, boilerplate is often produced with high confidence, so confidence should rank generic comments too high, for the same reason $S$ alone does. The comparison across the three comment types described next tests this directly.

Proposed Experiments

The primary testbed is the GOODPOINT-ICLR temporally held-out ICLR 2026 test set. Author responses there give per-comment labels for validity (the author agrees) and actionability (the author commits to a revision). We use $K=8$ prompt variants built from ProSA's templates and LLMCrit demonstration orderings, and $M=4$ counterfactual edits per claim. We report area under the F1–coverage curve and the Pearson correlation between score and validity. We also report author-validity at fixed coverage for the filtered reviews.

The secondary testbed is the SWIF²T 300-example paragraph-level test set, drawn from NLPeer, PeerRead and ARIES. There we rerank toward a final review and repeat the pairwise human helpfulness protocol on its 100-example human subset. LLMCrit's ICLR-introduction and Python-code domains test whether the pattern transfers beyond peer review.

Three checks are decisive. The first is a three-way comparison of comment types. On a sample of 300 comments hand-labelled as valid-specific, generic, or hallucinated, the valid comments must occupy the high-$S$, high-$D$ region. Generic comments must cluster at high $S$ and low $D$. If generic comments instead show low $S$, the two-axis story collapses into SENSEL. The second check is that the product must beat $S$-only, $D$-only, the embedding proxy, and token confidence by a margin larger than the variation across random seeds. The third is a cost check. We report how much of the gain survives with $K=2$ demonstration orderings and swap-only edits. The method is only useful if a cheap version sits well below SWIF²T's seven-minute latency while keeping most of the improvement in validity.