Overruling Without Residue: Contraction Discipline for Hierarchical Peer Review

When a reviewer argues that a paper's claim is unsupported, and an area chair later overrules that reviewer, the paper's standing should return to exactly what it would have been had the objection never been made. For a record that tracks which claims depend on which results, this is surprisingly hard to guarantee. Suppose the record is kept in a lightweight logic, meaning one without "or" and without general negation, of the kind used for tractable ontologies and rule systems. Belief-revision theory has shown that in such logics, removing a belief and then re-adding it does not restore what was lost. This property is called the recovery postulate, and it fails there Inconsistencies, Negations and Changes in Ontologies. Every contraction erases some neighbouring claims that cannot be rebuilt from the re-added one.

A multi-tier editorial board makes this failure concrete. Reviewers act first, area chairs overrule some of them, and editors overrule some area chairs. If each tier edits the shared record as it goes, every overruled objection leaves permanent damage. We propose a specific fix and test it. The shared record never changes. Each tier contributes only withdrawal requests, and the live state is always recomputed as one simultaneous withdrawal of all currently upheld requests from the original record. We then ask two questions. When may lower tiers safely apply their edits immediately? Does keeping this discipline measurably reduce the fragility that LLM-simulated review panels are known to show?

Core Claim

Recovery cannot be rescued inside the logic. It can, however, be replaced at the organisational level by a property we call overrule-invariance. Let $B$ be the original claim base of a paper, written in Horn form: each rule reads "if these results hold, this claim holds," with no "or." Let $R$ be the set of withdrawal requests that currently stand, and let $W(B,R)$ be a withdrawal operator that removes all of them at once. Overrule-invariance requires that overruling request $r$ yields exactly

$$W(B, R\setminus\{r\}),$$

the state that would have existed had $r$ never been filed. Recomputing from $B$ satisfies this by construction. The research content lies elsewhere, in three places. First, the simultaneous operator must itself be well behaved in Horn logic: it should not depend on the order of requests, and it should satisfy the Relevance postulate, which says it removes only what some upheld request actually needs removed. Second, we characterise when eager, tier-by-tier application coincides with recomputation, which is the cheap and interactive regime that real boards and agent systems use. Third, we test whether a paper-level claim record kept under this discipline makes decisions less sensitive to a single biased reviewer or to the area chair's style.

Relation to Prior Work

The belief-change work supplies both the constraint and the tools. On the constraint side, Contraction on the Semantic Web: On the Role of Relevance and Recovery proves that no major web ontology language satisfies recovery, while all of them admit Relevance-compliant contraction. Rational Elimination of DL-Lite TBox Axioms shows the over-erasure mechanism directly: without disjunction, an axiom is either kept whole or deleted. We accept both results as fixed and do not try to recover recovery.

On the tools side, we reuse three pieces. Entrenchment-Based Horn Contraction gives single-request Horn withdrawal driven by a priority ranking. DL-Lite Contraction and Revision supplies the polynomial-time and incoherence-avoiding design principles. Concept Contraction in the Description Logic EL warns that intersecting all candidate remainders yields trivial results, which rules out the naive "full meet" choice for our simultaneous operator.

All of these treat a single agent issuing one request at a time. The closest multi-request work is Parallel Belief Contraction via Order Aggregation, which contracts several items at once by aggregating preference orders. It differs from our setting in two ways. It works in full propositional logic, where recovery holds. It also imposes Parity, meaning all inputs are treated symmetrically. We borrow its intersective idea but work in Horn logic, and we let tier rank break ties, which Parity forbids.

The governance side motivates the problem and provides the testbed. AgentReview: Exploring Peer Review Dynamics with LLM Agents simulates ICLR review with reviewer and area-chair agents. It finds that one biased reviewer changes the decision in 37.1% of cases, and that switching the area chair from Authoritarian to Conformist shifts 30.6% of decisions. It keeps no record of which claims survive review, so it cannot say whether an overruled objection left residue. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that designated hierarchy helps LLM teams, but its agents keep no symbolic state. ResearchTown: Simulator of Human Research Community models research communities as monotone message passing, with no retraction at all.

Research Program

The formal part has three steps.

  1. Defining the operator. We define a simultaneous Horn withdrawal for this setting. For each request, we compute its kernels, the minimal subsets of the base that still entail the challenged claim. We then choose one "cut" per kernel, guided by a priority ranking on base rules and by the rank of the tier that upheld the request. We prove that the result does not depend on request order, satisfies Relevance, and is overrule-invariant when recomputed from $B$.
  2. When eager editing is safe. We then study eager application, where each tier contracts the current state as soon as it acts. Our conjecture is that eager and recomputed states coincide under two conditions. The first is that the kernels of distinct requests share no base rule. The second is that the cutting choice for one kernel does not depend on which other kernels are present. When kernels overlap, eager application can over-erase. We will bound the excess by the number of shared rules, and give a cheap test that tells a lower tier when it must defer instead of committing.
  3. Cost. We analyse the cost of recomputation. The concern raised for the critical-axiom approach of Rational Elimination of DL-Lite TBox Axioms is the need for logical closure. That concern is mild here, because a paper's claim base holds tens to low hundreds of rules, not an ontology the size of SNOMED CT.

The empirical part turns real review threads into this setting. For each paper, an LLM extractor produces a Horn claim base: experimental results as facts, and "result A and result B support claim C" as rules. It maps each reviewer weakness to a withdrawal request against a claim or a supporting rule. It maps each area-chair meta-review to a verdict on each request: upheld or overruled. Extraction is the weakest link, so 100 papers will be hand-annotated by two annotators, and all headline metrics will be reported on that subset as well as on the automatic set.

We then run the AgentReview pipeline twice. In the first run, agents deliberate in free text as published. In the second, every agent action that challenges a claim must be filed as a request against the shared base. The area chair and final decision agents see the recomputed surviving claim set, in addition to the text.

Baselines

The naive reference point is the unmodified AgentReview pipeline AgentReview: Exploring Peer Review Dynamics with LLM Agents. It is run on the same 523 ICLR submissions it was built on, with the same model and the same persona grid. This is also the strongest existing method on our question, because it is the only system in the file that models tiered review with overruling. Its reported fragility numbers set the bar: the decision changes in 37.1% of cases when one biased reviewer is added, and 30.6% of decisions shift between Authoritarian and Conformist area chairs. We re-measure both under our compute and prompts rather than relying only on the published figures. Beating this baseline means a statistically significant drop in both flip rates. It also means no loss on the paper's fidelity check, where agents identified the same major strengths and weaknesses as humans in 90% of cases.

The second baseline is eager sequential Horn contraction using the entrenchment operator of Entrenchment-Based Horn Contraction. Each tier edits the current state immediately, and an overrule is handled by re-adding the claim. This isolates recovery failure. It uses the same claim bases, rankings and agent prompts as our method, so any gap between the two is attributable to recompute-versus-eager alone.

The third baseline is snapshot rollback: when any request is overruled, revert to the base as it stood before that request, then re-apply later requests in their original order. It gets the same bases and prompts.

The fourth is flat majority aggregation over requests, which removes the tier structure entirely.

The key ablation keeps the structured claim list and the request interface but drops the support rules. A withdrawn claim then disappears alone, with no dependency reasoning. This tells us whether any gain comes from the logic or merely from forcing agents to name claims.

Why not just…?

Why not just keep a version history and roll back? This is the most serious alternative, and in the simplest case it is exactly right. Our recompute rule is essentially principled rollback. Rollback goes wrong in the interleaved case. Reviewer 2's legitimate withdrawal was filed after reviewer 1's later-overruled one, so reverting to before reviewer 1 either discards reviewer 2's work or re-applies it in an order that, in Horn logic, gives a different state. The snapshot baseline measures how often this happens on real threads. If interleaved overrules with overlapping kernels turn out to be rare, the formal contribution reduces to "replay the log." We would report that honestly as the finding.

Why not just let LLM agents deliberate in free text? That is the published pipeline, and its documented sensitivities (37.1%, 30.6%, and an 18.7% engagement drop caused by one irresponsible reviewer) are the problem we target. Free text gives no way to check whether an overruled objection still colours the area chair's summary. Our residue metric measures exactly that: the claims still treated as withdrawn after their only upheld support was restored.

Why not just use full first-order logic, where recovery holds? Paper claim bases do not need that expressivity. Classical contraction would also keep synthetic disjunctions such as "claim C or result A," which have no editorial meaning and cannot be shown to authors. The tractability argument from DL-Lite Contraction and Revision matters less at this scale, so the real reason for rejecting this alternative is interpretability, not cost.

Why not just take a majority vote over reviewers' objections? This removes overruling altogether. It also inherits the conformity collapse AgentReview reports, a 27.2% fall in rating variance during discussion. The flat baseline tests whether tier-ranked cutting adds anything beyond counting.

Proposed Experiments

The first decisive check is residue. On the hand-annotated subset, we measure the fraction of papers where, after all overrules, the eager baseline's surviving claim set differs from the recomputed one. We also count claims wrongly lost per paper. Our conjecture predicts that such differences occur only when kernels overlap. Verifying this per paper tests the characterisation directly.

The second check is fragility. We inject one biased reviewer into the AgentReview persona grid, and separately toggle the area-chair style, then measure decision flip rates for all four systems and the ablation.

The third check is fidelity. We compare surviving claim sets with the claims human meta-reviews endorse, using annotator agreement as the ceiling.

For held-out validation beyond the model's training cutoff, we apply the extractor and pipeline to the 200 review tasks in ReviewBench ulab-ai/research-bench. ResearchTown reports a semantic similarity of 0.49 to human reviews on this set. We use that figure only as a note on how hard the text is to simulate, not as a target, because our metrics concern claim sets.

A negative result on the fragility check would also be informative. It would suggest that area-chair style effects are driven by framing, not by lost claims.