AI Reviews Are Biased Towards the Appendix

Saptarshi Ghosh, Mohammad Saim, Tianyu JiangarXiv 2026 · arXiv

TL;DR

Modern Large Language Models (LLMs) like GPT-6 and Claude 5.5 have developed a significant "appendix bias," drawing over 50% of their peer-review criticisms from supplementary materials rather than the main paper. This behavior is now leaking into human-written reviews, where appendix-related weaknesses have nearly tripled in the last two years, potentially shifting the focus of scientific evaluation from core ideas to technical minutiae.

What Problem Does This Paper Address?

As LLMs become ubiquitous in the research workflow, they are increasingly used to "co-pilot" the peer-review process. While previous studies have looked at how well LLM reviews align with human scores, we know very little about where these models look for flaws.

In traditional peer review, the "main body" is the star—it contains the core logic, methodology, and primary results. The appendix is a supporting actor, reserved for hyperparameter tables, extended proofs, or niche experiments. If LLMs are disproportionately obsessed with the appendix, and human reviewers are increasingly relying on LLM-generated drafts, we risk a "tail wagging the dog" scenario where papers are judged more on their supplementary data than their fundamental contributions.

Methods

The authors developed a two-step pipeline to analyze the spatial bias of AI reviews:

  1. Review Generation: They prompted four state-of-the-art models (GPT-6-Astra, GPT-6.1-Sol, Claude-Opus-5.5, and Claude-Sonnet-5.5) to provide "ICLR-style" reviews for 2,000 anonymized papers from major AI conferences (ACL, EMNLP, COLM, and ICLR).
  2. Section Attribution: They then asked the models to map each generated weakness back to specific sections or page numbers in the original PDF.

To ensure the results weren't just an artifact of how the models "read" files, they tested multiple access methods, including direct API calls, coding agents (Codex), and standard web interfaces. Finally, they compared these AI reviews against a historical baseline of human-written reviews from ICLR 2021–2027.

Figure 2: Pipeline of our method: i) Extract and sample papers from four conferences with a minimum appendix length of 15 pages, ii) Generate an AI review, and iii) map each weakness back to the section it belongs to.

Datasets

The study utilized a dataset of 2,000 research papers:

  • Recent Papers: 800 papers from ACL 2026, EMNLP 2026, COLM 2026, and ICLR 2027 (chosen to ensure models hadn't seen them during training).
  • Historical Baseline: 1,200 human-reviewed papers from ICLR 2021–2026 to track how human reviewing habits have changed over time.
  • Criteria: All sampled papers were required to have an appendix of at least 15 pages to provide enough "surface area" for the models to criticize.

Key Insights and Ideas

  • The "Flat" Reading Bias: Unlike humans, who weigh the main body more heavily, LLMs treat the entire document as a single, flat unit. A missing confidence interval on page 35 of the appendix is treated with the same severity as a logical flaw in the introduction.
  • Auditors vs. Reviewers: LLMs act more like technical auditors than scientific reviewers. They excel at cross-checking tables and demanding more statistical validation (more seeds, more ablations) but rarely engage with the "big picture" framing or the core motivation of the work.
  • The Leakage Effect: The most striking insight is the evidence of AI "contamination" in human reviews. As shown in Figure 1, human reviewers historically cited the appendix in only 6–10% of their weaknesses. After the release of GPT-6 and Claude 5.5, this number skyrocketed to 27.60%.
Figure 1: Newer models such as GPT-6-Astra and Claude-Opus-5.5 are biased toward the appendix when writing reviews. They draw more than 50% of their weaknesses from appendix content. This bias now seems to be showing up in human reviews. For years, human reviewers at ICLR cited only about 6-10% of their weaknesses in the appendix. This share began to rise after the release of GPT-5, and nearly tripled to 27.60% in ICLR 2027 reviews, posted shortly after GPT-6-Astra and Claude-Opus-5.5 were released.

Key Results

  • AI Obsession: GPT-6-Astra and Claude-Opus-5.5 draw more than 50% of their weaknesses from the appendix. In some models, this behavior is consistent regardless of whether they access the paper via API or a web chat interface.
  • Human Shift: In ICLR 2027 reviews, the percentage of human-written weaknesses targeting the appendix reached 27.60%, nearly tripling the historical average.
  • Spatial Distribution: Humans focus their criticism on the first five pages (the core idea). LLMs almost entirely ignore the opening pages and instead peak in their criticism around pages 6–7 (results) and deep into the appendix (pages 20+).

Limitations and Future Work

  • Attribution Reliability: The study relies on LLMs to attribute their own weaknesses to sections. While the authors manually verified a sample, there is a risk that models might "hallucinate" which section a weakness belongs to.
  • Correctness vs. Location: The paper focuses on where the criticism comes from, not whether the criticism is correct. An appendix-based weakness might be a valid "catch" that improves the paper, even if it's minor.
  • Causality: While the correlation between LLM releases and the shift in human reviews is strong, it is not definitive proof of causality. Future work could involve controlled user studies to see if reviewers explicitly move toward appendix-heavy criticism when prompted by AI.

Summary updated

Abstract

Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. This behavior holds across the API, coding-agent, and web access methods for using LLMs to review a paper. In contrast, we find that humans draw only 8.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper's presentation and content related to its core idea. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine-grained technical details not central to the paper's contributions. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews. Our code is publicly available at: https://github.com/cincynlp/review_appendix.
BibTeX
@misc{ghosh2026ai,
  author = {Saptarshi Ghosh and Mohammad Saim and Tianyu Jiang},
  title = {{AI Reviews Are Biased Towards the Appendix}},
  year = {2026},
  eprint = {2611.17811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17811},
}
Figure