Ghost Perspectives: Detecting Memorized Reviewer Opinions in Multi-Perspective Summarization

Some summarization tasks have no single right answer. Three peer reviewers of the same paper highlight different strengths and flaws, and a forum thread holds competing answers. The benchmarks built for this "multi-perspective" setting draw on public web text such as OpenReview papers with their reviews and Stack Overflow or Yahoo! Answers threads. Large language models were almost certainly trained on that text. A model asked to summarize a 2020 ICLR paper "from a reviewer's point of view" may not need to read the paper at all. It may simply recall what the real reviewers said. Existing contamination detectors look for memorized strings. Here the memorized thing is an opinion: which weaknesses reviewers raised, which answer a thread converged on. That can be reproduced in fresh wording that no string-matching test will flag.

We propose to detect this through counterfactual edits that invalidate a specific reviewer point. Take a criticism a real reviewer made, such as "the paper lacks an ablation of component X". Edit the source paper so the criticism is no longer true, for example by inserting that ablation. A model that reads the paper should stop raising the point. A model that is echoing memorized reviews will keep raising a criticism that the document in front of it no longer supports. We call these leftover criticisms ghost perspectives. The fraction that survive the edit gives a contamination signal that does not depend on knowing a model's training cutoff and does not need the model's token probabilities. We validate the measure by deliberately contaminating an open model with known review text. We then use it to test an open question: whether memorization helps cause the "perspective collapse" that has stalled this field.

Core Claim

In multi-perspective summarization, contamination appears as perspective recall rather than verbatim copying. The claim is that a targeted counterfactual edit exposes this recall more reliably than three existing tools: date-based splits, membership-inference tests, and generic paraphrase perturbations. By membership inference we mean tests that ask, from a model's token probabilities, whether a text was in its training data.

The gap matters for two concrete reasons. First, the main scientific benchmark, MuP, built from roughly 12,000 ICLR, NeurIPS and AKBC papers with about 2.57 reviewer summaries each, has never been audited for pre-training exposure (Overview of the First Shared Task on Multi Perspective Scientific Document Summarization (MuP)). The same holds for AnswerSumm's roughly 96,701 forum threads (Multi-Perspective Abstractive Answer Summarization). Second, the MuP organizers reported that every submitted system collapsed to one generic summary instead of distinct per-perspective outputs. Nobody has asked whether memorized, high-frequency text such as the paper's own abstract or the dominant review contributes to that collapse.

Relation to Prior Work

Three papers sit closest to this proposal.

Benchmarking LLMs' Judgments with No Gold Standard builds GRE-bench from fresh ICLR 2023 reviews to dodge leakage. It scores generated reviews with GEM-S, an estimate of how much information a generated review carries about the human reviews beyond what the abstract already gives. GEM-S removes credit for regurgitating the abstract. It does not remove credit for regurgitating memorized reviews, and it scores pooled reviews rather than separate perspectives. We reuse it as a scoring baseline and test whether it is fooled by ghost perspectives.

CAP: Data Contamination Detection via Consistency Amplification supplies the core logic: memorized behavior is brittle under perturbation. It applied that logic to single-reference financial summarization (ECTSum). Its perturbations are task-generic, such as paraphrasing or shuffling distractors, and do not specifically change which perspective is correct. Our method differs in one ingredient: the edit is aimed at a known reviewer claim. We isolate this ingredient with an ablation.

FABLES avoids contamination by using only books published in 2023 or 2024 and checking faithfulness with human readers, at about $5.2K for 26 books. That makes it a temporal avoidance strategy for single narratives rather than a detector.

Two further results limit which designs are worth trying. Test of Time and Beyond Memorization show that the drop in scores after a model's cutoff date is fragile. Cloze questions show a -3.83 point post-cutoff drop. Reasoning questions built from the same papers show none, and physics scores even rise by +2.19 points. Influence-function tracing back to the source document falls from a 77.5% to a 17.5% top-1 hit rate once the text is reworded. Both results imply that date splits and string-level tracing will under-detect reworded opinions. That is why we do not rely on either. Data Contamination Calibration for Black-box LLMs (PAC) supplies our strongest membership-inference competitor.

Research Program

Building the probes. For each paper in MuP, and in post-cutoff review corpora, we take the full public reviews. From them we pull out "checkable" claims: statements about the paper's content that a targeted edit could make false. Examples include a missing baseline, a missing ablation, a stated dataset size, or an unacknowledged limitation. An LLM drafts a minimal, internally consistent edit to the paper for each claim. The edit inserts the missing ablation table with plausible numbers, or adds the requested baseline. Humans verify a stratified sample of these edits for consistency.

We also build three kinds of controls:

  • Neutral edits of matched length that touch no reviewer claim.
  • Generic edits, which paraphrase the whole paper in the style of CAP.
  • Reverse edits, which delete content so that a criticism no reviewer raised becomes valid. These test whether the model reads the paper at all.

For AnswerSumm the analogue is to delete an answer that carried a distinct perspective in the reference bullet points. A ghost perspective is then a summary bullet that still reports the deleted viewpoint.

Measuring ghost perspectives. We ask each model for per-perspective summaries of the original document and of the edited one. For each invalidated claim, a narrow question is posed to a natural-language-inference model (one that checks whether a text implies a statement) and, separately, to an LLM judge: "does this summary assert claim c?" The yes/no framing avoids the weakness that open-ended LLM judging shows when the judge cannot itself solve the task, a weakness LiveBench documents and FABLES confirms, with a 58.2% F1 auto-rater. The ghost rate is the fraction of invalidated claims that are asserted after the edit, counted only among claims the model asserted before it. The contamination signal is

$$\Delta_{\text{ghost}} = G_{\text{exposed}} - G_{\text{clean}},$$

where $G_{\text{exposed}}$ is the ghost rate on documents whose reviews the model may have seen, and $G_{\text{clean}}$ is the ghost rate on matched documents it cannot have seen. Subtracting $G_{\text{clean}}$ removes the base rate of ghosts that come from inattention or long-context failure rather than memory.

Ground truth by deliberate contamination. Detection claims are only credible against known labels. We take an open-data model such as OLMo2-7B-Instruct, the model used in the influence analyses of Test of Time. We continue pre-training it under four conditions:

  • papers together with their reviews;
  • papers only;
  • reviews only;
  • no injection.

All conditions use a split of post-cutoff ICLR 2024 and NeurIPS 2024 papers, drawn for example from the ReviewBench portion of ResearchBench. These conditions give exact labels for which reviews each model has seen. They also separate memorizing the paper from memorizing opinions about it.

The collapse question. In the same controlled models we measure perspective diversity. One measure is the Semantic Area embedding-volume score from the AnswerSumm work. The other is the number of distinct reviewer claims covered per summary set. This tests whether injected reviews or injected abstracts push outputs toward a single consensus summary. Seeing reviews might instead reproduce diversity that was merely memorized, which would be the opposite of collapse. Either outcome bears on the MuP organizers' unexplained finding.

Baselines

Naive reference point. Leaderboard scores are taken at face value with no contamination analysis. On MuP the strongest reported system is the AINLPML extract-then-rewrite pipeline. The overview reports human readability 4.7 and faithfulness 4.4, and the team's paper reports gains of 0.28 ROUGE-1 F and 0.71 average ROUGE F over the organizer baseline (Team AINLPML @ MuP in SDP 2021). On AnswerSumm the reference is BART with multi-reward reinforcement learning at an NLI score of 56.99. Our audit asks how much of such scores survives once ghost-prone items are separated out.

Temporal split. This compares scores on pre-cutoff MuP with matched post-cutoff papers, drawn from ICLR 2024 and NeurIPS 2024 reviews and from the 2023–2025 multi-round Nature Communications reviews of FIRSTPASS. It is the simplest competitor. It is run on the same models and the same summary prompts.

Membership inference. We run Min-K% Prob, perplexity, and PAC, the strongest of the three. PAC reports +4.5% detection AUC on WikiMIA and +5.9% on StackMIA over Min-K% and perplexity (Data Contamination Calibration for Black-box LLMs). All three are applied to review texts and paper texts, using the token-probability tricks from that paper for API models. No reported result exists on Lacuna for any of these methods on MuP, AnswerSumm, or peer-review text.

CAP-style generic perturbation. We compute CAP's Performance Consistency Ratio under paraphrase and fact perturbation. The perturbation budget is matched to ours in the number of edits per document. No reported result exists on Lacuna for CAP on multi-perspective benchmarks.

GEM-S as a scorer. We test whether GEM-S assigns lower scores to summaries containing ghost perspectives.

Fairness and the success criterion. Every detector gets the same documents, the same models, and the same compute budget per document. Any detector thresholds are tuned on a shared validation split of the injected models. On the controlled models, "beating" a baseline means higher detection AUC for which reviews were seen. The hardest case is the reviews-only injection, where the paper text itself was never seen, so membership tests on the paper should fail by construction. Beating PAC means exceeding its AUC by more than the confidence interval from bootstrap resampling over papers.

Ablation. Replacing targeted edits with neutral edits and with generic paraphrase isolates the key ingredient. If generic perturbation recovers most of the AUC, our contribution reduces to CAP applied to a new task.

Why not just…?

Why not just evaluate on papers published after the cutoff? We will, and it serves as a baseline. But it only avoids contamination; it does not measure it on the benchmarks the field has used for years. Test of Time also shows that the post-cutoff drop appears or vanishes depending on how a task is phrased. Cutoffs of closed models are estimates, and later fine-tuning can leak newer data. The deliberate-contamination models settle the question directly: if the date split separates injected from clean papers as well as $\Delta_{\text{ghost}}$ does, the simpler method wins on those models, and we will say so.

Why not just run membership inference such as PAC on the reviews? This is the most serious competitor when reviews were seen verbatim, and on some models it may match us. Its expected weakness is the reviews-only and reworded-opinion cases. Influence tracing drops to a 17.5% hit rate once text is reworded, according to Test of Time, so token-probability signals on the original strings should weaken in the same way. The four-condition injection experiment, plus a condition where only LLM-paraphrased reviews are injected, will show exactly where PAC fails or holds. Membership inference also says whether a text was seen, not whether that exposure changes the summaries. Ghost rate measures the effect on the output directly.

Why not just apply CAP's generic perturbations? CAP's paraphrases leave every reviewer claim true. So a model that recalls opinions can stay perfectly consistent under them and still be contaminated. The targeted-versus-generic ablation tests this directly. If generic perturbation performs equally well, we will report the method as CAP extended to multiple perspectives rather than as a new probe.

Why not just ask GPT-4o whether the summary covers the reviewers' points? Coverage judged against the original reviews rewards exactly the recall we want to detect. LLM judges are also inflated by padding, and GPT-4o Examiner reached only 0.537 Spearman correlation with instructor grades in the GRE-bench work. Our judge answers only a yes/no question about one named claim. We check it against human labels on a sample before trusting it.

Proposed Experiments

The first decisive check is validation on the injected models. If $\Delta_{\text{ghost}}$ is near zero for the papers-plus-reviews and reviews-only conditions, the core claim fails. The claim also fails if PAC and the date split detect those conditions equally well. The neutral-edit control must show no ghost gap. The reverse-edit control must show that every model registers newly introduced flaws. Without both, ghost rate measures nothing beyond reading ability.

The second is an audit of existing models on MuP and AnswerSumm. The planned set is open models including Llama-3.1, plus API models (GPT-4o, Claude-3-Opus) at their default settings. We report the ghost gap against a FIRSTPASS and 2024-OpenReview clean pool, matched on paper length and topic. We then re-rank the systems after removing items prone to ghost perspectives. The test is whether the ranking changes and whether reported coverage scores shrink.

The third is the collapse test. Within the controlled models, we compare perspective diversity across the four injection conditions. If injecting abstracts lowers diversity relative to no injection, while injecting reviews raises diversity together with the ghost rate, that pattern would explain why MuP systems collapsed. It would also show that high diversity on leaked benchmarks can be memorized rather than reasoned. A null result is also informative: it would mean the MuP collapse has another cause, pointing to training objectives rather than data exposure.

Human verification covers about 10% of edits and judge decisions. With roughly 2,000 targeted edits, that keeps cost well below FABLES-style full annotation, because each human check is one short claim rather than a whole document.