Arbitrating Whose Standard Applies: Editorial Tiers Against Dogmatic and Noncommittal Review

When a manuscript spans two fields, its reviewers often disagree about what counts as sound evidence, not about the evidence itself. A physicist may want a first-principles model. A biologist may want replicate controls. Automated reviewing systems built from language models fail here in two opposite ways. Sometimes a reviewer agent confidently rejects the work by the standards of the wrong field; call this the dogmatic failure. Sometimes the system hedges, averages, or refuses to commit because every standard looks equally arguable; call this the noncommittal failure. The philosophical literature names the collision of these two poles "epistemological double nihilism." Its proposed exit is a prudent middle: commit to a judgment, but say what would overturn it.

This note proposes to build and test one concrete version of that exit. An editor tier, played by a language model, does not average reviewer verdicts. Instead it does two things. First, it labels each reviewer objection by whose standard it invokes: the manuscript's home field, or a field the manuscript is not trying to satisfy. Second, it must issue a committed decision together with an explicit condition that would change it. We test this on a real multi-field, multi-round review corpus. Ground truth is what editors and later review rounds actually did. The question is whether this step reduces both failure modes at once, or whether, as prior work warns, an extra layer of model judgment simply rewards fluent-sounding reviews.

Core Claim

The useful work of an editorial hierarchy on cross-field manuscripts is not aggregation but arbitration. The editor decides which disciplinary standard an objection rests on, and whether that standard binds this manuscript. We claim that an editor agent required to perform this "standard attribution" and to state a falsifiable "what would change my decision" condition will do three things. It will reduce confident rejections driven by off-field standards. It will reduce noncommittal outcomes. And it will do both without lowering agreement with real editorial outcomes. A second, adversarial claim is part of the test. If the editor is vulnerable to the fluency bias documented for model judges, the hierarchy will amplify that bias, and we expect to see it.

Each direction plays a different role. Institutional silos supply the failure mechanism. Biomedical data-science careers stall because promotion and review panels judge hybrid work by single-discipline criteria (The Training of Next Generation Data Scientists in Biomedicine). Professional frameworks likewise enforce boundaries rather than reward integration (An action research protocol to strengthen system-wide inter-professional learning and practice). Editorial board structure supplies the intervention point. Journals already split work between a chief editor who sets global standards and associate editors who run domain-specific cascades (IEEE Transactions on Cybernetics publication information). Double nihilism supplies the evaluation lens: two named, opposite failure modes, both of which must be measured.

The philosophical sources motivate that lens; they do not provide the method. Batey contrasts the craving for puzzle-like single answers with Kafka's unresolved bureaucracy (Da Vinci versus Kafka). Balazs reads skepticism as an active "ethics of prudence" compatible with firm commitment (Comment peut-on être sceptique?). The closest thing to an empirical signature comes from a survey of 120 students using a 50-item critical-thinking instrument. It found high inquisitiveness (3.78) but low cognitive maturity (2.81) and a truth-seeking score of exactly 3.00 (A Study on the Current Situation of English Majors' Critical Thinking Skills). That pattern is open-mindedness without the willingness to commit. We take only the idea that the two failures are separately measurable. The single-site, 83% female sample cannot support more.

Relation to Existing Work

Three systems sit close to this idea, and each lacks a different piece.

FIRSTPASS collects 3,668 transparent-review records from Nature Communications across biology, chemistry, neuroscience, physics and earth science. It includes round-1 reviews, rebuttals and round-2 reviews. It predicts whether a paper needed standard or extended review. It treats the editorial outcome as a classification label and never models how an editor reconciles reviewers from different standards. We reuse its data and labels. We add the arbitration step, and we add measurements that its accuracy number cannot show.

PeerArg turns reviews into support/attack argument graphs and aggregates them with a fixed formal rule. It beats an end-to-end model on both ICLR 2018 reviews and medical-journal reviews. Its aspect nodes such as "soundness" are single universal scalars. That universality is exactly the assumption our standard-attribution step removes.

Multi-agent peer review collaboration shows that critiques from other agents prevent the self-correction collapse of single models. It uses a flat group with majority vote on closed-form puzzles. It has neither an editor tier nor distinct disciplinary criteria.

Work on organized agent teams supports adding a tier at all, though in very different settings. Hierarchical leadership improves embodied household coordination (Embodied LLM Agents Learn to Cooperate in Organized Teams). "Monarchical" agent societies outperform voting ones on visual question answering (Mindstorms in Natural Language-Based Societies of Mind). Neither involves contested evaluative standards, so we treat these results as weak priors, not evidence.

The paper that most constrains the design is Articulate Intuition or Genuine Analysis?. It shows that a model judge rating 3,563 ML reviews confused analytical form with analytical function. Review length explained about 74% of the apparent advantage of AI-written reviews. The judge also collapsed nine rubric dimensions into one factor. This directly threatens our editor agent, which is itself a model judge. We therefore make fluency sensitivity a primary outcome rather than an afterthought. We also borrow that paper's matched-probe design, which separated genuine fault-finding from surface fluency in 33 of 35 pairs.

Research Program

We build a two-tier review pipeline on FIRSTPASS manuscripts with discussion and conclusion sections stripped, following FIRSTPASS's guard against conclusion bias.

The lower tier consists of reviewer agents, each instructed with the evidentiary norms of one of the five fields. We derive these norms from the content of real reviews in that field, not from hand-written stereotypes.

The editor agent receives the manuscript and all reviews and must produce three things. The first is a standard attribution for each objection: home-field, off-field-but-relevant, or off-field-and-inapplicable. The second is a committed outcome, either standard or extended review, with no abstain option. The third is a short condition list stating what the authors would need to show for the decision to flip.

Forcing commitment could simply hide the noncommittal failure. To catch that, we measure hedging in the decision rationale and calibration of the editor's confidence.

To create controlled silo conflict, we build cross-field swaps. A manuscript from field A is reviewed by its own-field agent plus an agent primed with field B's norms. The real human reviews tell us which objections the home field actually raised.

We also use a naturally hybrid subset. These are manuscripts whose real round-1 reviews invoke methods from more than one field. We identify them with a tagger checked against about 200 objections labeled by domain experts.

The multi-round structure gives us a test that a single-round dataset could not. The editor's stated flip conditions can be scored against the concerns that the real rebuttal and round-2 reviews actually addressed. We call this condition recall. It asks whether the editor identified the doubts that mattered, not only whether it guessed the label.

Baselines

Naive references. The first reference is the majority class. Following FIRSTPASS's reported 63–71% standard-review rate, this sets the accuracy floor. The second is a length-only classifier on concatenated review text. It is cheap, and the 74% length finding suggests it may be surprisingly strong. Any method that does not clearly beat the length classifier on the robustness metrics has learned nothing about standards.

Strongest existing method. This is the FIRSTPASS fine-tuned Qwen2.5-7B-Instruct outcome predictor. It reports 80.5% accuracy and 78.2% macro-F1 on outcome prediction, outperforming a zero-shot Gemini flash-lite model by 10.4 points (FIRSTPASS). We rerun it on our exact splits. Our editor uses the same Qwen2.5-7B base, fine-tuned on the same training data, plus one larger open model for both systems.

We do not expect to beat it by a wide margin on in-distribution accuracy, and we do not need to. Beating it means two things. On standard splits, staying within its confidence interval on macro-F1. On cross-field swaps and the hybrid subset, showing a significantly lower rate of decisions flipped by an inapplicable off-field objection.

Argument-graph baseline. For PeerArg we reimplement its argument-graph aggregation on FIRSTPASS reviews with the same base model. Its reported 68.09% macro-F1 on medical-journal reviews, against 59.5% for an end-to-end Mistral-7B, is the clearest cross-field result in the file.

Flat persona ensemble. We run the Xu et al. creation–review–revision protocol with our discipline-specific reviewer agents and majority vote. It gets a matched token budget.

Single-prompt arbitration. One model receives all reviews and the same attribution-and-commitment instructions in a single prompt. Its token budget is matched to the full pipeline through self-consistency sampling.

Ablations. Two ablations isolate the key ingredient. The first removes standard attribution while keeping the editor tier. The second removes the flip-condition requirement.

Fairness rules. All systems share data, base models, a fixed tuning budget of the same number of prompt or hyperparameter trials, and the same total inference tokens per manuscript.

Why not just…?

Why not just use one strong model as meta-reviewer? This is the most serious threat. It is why single-prompt arbitration is a baseline rather than a strawman. Two pieces of evidence count against the naive version without attribution instructions. PeerArg's end-to-end baseline trailed structured aggregation by about 8.6 points on medical reviews. And the model judge in the Kahneman4Review work was dominated by length. But if single-prompt arbitration matches the two-tier pipeline on every robustness metric, we will conclude that the tier is incidental and the attribution step carries the gain. We will report the result that way; it remains a useful finding.

Why not just ensemble discipline-specific reviewers and vote? Voting is exactly where the dogmatic failure should appear. If two of three agents hold the off-field standard, the vote rejects. Zhuge et al. report that voting societies pool shallow majority biases. Our cross-field swap varies the number of off-field agents from one to two. If the vote's error rate on inapplicable objections does not rise with that number, this alternative is vindicated.

Why not just use the fine-tuned FIRSTPASS classifier? It may already be the most accurate predictor. However, a label offers no way to tell whether a correct decision came from the right standard. It also cannot produce flip conditions to score against round 2. The swap experiment tests whether its accuracy holds when off-field objections are injected. If it does hold, the practical case for arbitration narrows to interpretability. We would say so.

Why not just control for review length? Length normalization would remove the cheapest fluency confound, so we apply it to all systems. It does not address empty reviews that are padded to normal length. Our matched probes target exactly that case.

Proposed Experiments

The first decisive check is the cross-field swap. For each FIRSTPASS manuscript we inject an off-field reviewer whose objections domain annotators mark as inapplicable. We then measure the dogmatic error rate: decisions flipped toward extended review by those objections. We measure the noncommittal rate over the same items: rationales judged as hedging by annotators blind to the system. The proposal succeeds only if the attribution editor lowers both rates relative to the persona vote and the attribution-free editor, while staying within the confidence interval of the FIRSTPASS classifier on unperturbed macro-F1. Lowering one rate by raising the other counts as failure.

The second check is the fluency probe, adapted from Kahneman4Review. We add matched pairs of reviews of equal length. One identifies a real methodological fault that the human round-2 review confirms. The other is fluent but contentless. We measure how much each system's decision shifts toward each. If the editor shifts more toward the empty review than the flat ensemble does, the hierarchy amplifies form over function. That would answer the open tension in the literature negatively, and we would report it as such.

The third check is condition recall. We score the editor's flip conditions against rebuttal and round-2 concerns using expert-validated matching. We compare against conditions extracted post hoc from the PeerArg and single-prompt baselines.

Finally, as a transfer test without retraining, we run all systems on the medical-journal reviews where PeerArg reported 68.09% macro-F1. This tells us whether field-norm reviewer agents learned from five natural-science fields generalize to a sixth.