Journals and conferences are starting to build review pipelines out of language-model agents. One agent plays editor, others play associate editors for particular subfields, and others write reviews. These pipelines copy the board structure of real journals, and that structure is an information-hiding device. Reviewers are not supposed to know who the authors are. One reviewer is not supposed to see another's verdict before writing their own. A subfield editor's private notes are not supposed to reach the authors. Each of these is a promise that some piece of information stays inside its silo. When agents pass free-form text to each other, nothing enforces that promise. An editor's summary can carry an affiliation cue downstream. A reviewer can paraphrase a colleague's score. The final accept/extend decision can then shift with information that should never have reached it.
This proposal treats each of those promises as a hyperproperty: a requirement that compares two runs of the system rather than inspecting one. "Author identity does not influence the review" means that two runs differing only in author identity must look the same, in distribution, at every point where a silo boundary is observed. We propose a runtime monitor for multi-tier LLM review pipelines. It checks this kind of distributional non-interference at every tier boundary, not only at the final output. It reuses shared run prefixes so that it spends model calls only downstream of where a secret first enters. It reports which tier caused the leak. The claim to test is concrete. On pipelines with deliberately planted leaks, tier-local paired monitoring should find leaks with fewer model calls, and find leaks that comparing final decisions misses, than the obvious end-to-end swap test.
Core Claim
A multi-tier review pipeline can be specified as a set of silo boundaries, each with a secret and an observable. A boundary might be "everything the reviewer agent receives," with the secret being author affiliation. Another might be "reviewer 2's first-round report," with the secret being reviewer 1's report. Checking the specification as probabilistic non-interference at each boundary is cheaper and more diagnostic than checking only the pipeline's end decision. Let $x$ be the public submission content, $s$ and $s'$ two values of a secret, and $O_b$ the text that crosses boundary $b$. The property we monitor is
$$ \forall x,\ \forall s, s':\quad D\big(P(O_b \mid x, s),\ P(O_b \mid x, s')\big) \le \epsilon_b , $$
where $D$ is a two-sample divergence estimated from sampled runs and $\epsilon_b$ is a tolerance set per boundary. With sampled language models, two runs almost never produce identical text. The classical deterministic form ("same low inputs imply same low outputs") is therefore useless here, and the distributional form is required. Two things are new. The first is monitoring this property per boundary inside a hierarchical agent pipeline. The second is the claim that boundary-level checks, combined with sharing of run prefixes, beat end-to-end counterfactual testing on both detection power and cost.
What Exists and What Is Missing
The formal tools come from the hyperproperty literature. Deciding Hyperproperties introduced HyperLTL, a temporal logic that quantifies over several runs at once, so non-interference becomes $\forall\pi\forall\pi'$ over pairs of traces. Efficient monitoring of hyperproperties using prefix trees showed that a runtime monitor storing observed runs in a prefix tree, so that work is shared until runs diverge, cut active monitor instances from nearly 2 million to 248 and checked 64-bit non-interference in 7.5 seconds. That paper supplies our core efficiency mechanism. Its monitors assume deterministic, symbolic traces, though, and we must replace equality checks with statistical tests. Probabilistic Hyperproperties with Nondeterminism and Parameter Synthesis for Probabilistic Hyperproperties verify hyperproperties of probabilistic models through self-composition, meaning running copies of the system side by side. They justify our distributional formulation, but they assume an explicit finite-state model, and an LLM pipeline has no such model. Hypercontracts gives assume-guarantee algebra for hyperproperties over component sets. We borrow its idea that a global property should follow from local per-component guarantees. We use it as an informal design principle and test it empirically rather than proving anything about it.
The closest connection between hyperproperties and multi-agent systems is HyPOLE. It turns $\forall\exists$ HyperLTL specifications into dense rewards for multi-agent reinforcement learning on StarCraft-style games, reaching a 60% win rate on the Corridor map where vanilla QMIX stays below 5%. It reports a 62% training-time overhead and is limited to discrete actions. HyPOLE uses hyperproperties to train agents. We use them to audit text-passing agents after deployment, and our "actions" are free-form messages.
The editorial side motivates the problem. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that leader-follower hierarchies make agent teams faster, improving completion time by 9.76% for three GPT-3.5 agents. The same authors note that such agents' behavior is hard to bound, and that gap is exactly what we address. ResearchTown simulates research communities, including reviewing, as message passing between agents. It gives us a ready substrate for pipelines. FIRSTPASS supplies real multi-round review trails (round 1, rebuttal, round 2, decision) from Nature Communications. Their tree shape is what makes prefix sharing pay off.
On the silo side, Permissioned Blockchain Technologies for Academic Publishing and Blockchain-based Enterprise Community Data Access Control Methods and Applications enforce silos by controlling who may read which record. They constrain our design by showing what access control does and does not cover. Access control decides whether an agent may read a record. It cannot tell whether an agent that legitimately read something then leaked it in its output.
Group Fairness in Peer Review plays a smaller role. It shows that standard reviewer assignment violates the stability condition known as the core in nearly 100% of runs, which gives subfields an incentive to secede. This motivates treating subfield tracks as silos with their own editors, which is the pipeline topology we audit.
Research Program
We first build a configurable hierarchical review pipeline. It has an editor agent, per-subfield associate-editor agents, and two or three reviewer agents. It runs a FIRSTPASS-style two-round process and ends in a decision that matches FIRSTPASS's label (standard versus extended review). Every message between agents is logged at a named boundary. A specification file lists, for each boundary, the secret it must not depend on. Examples are author name and affiliation, the other reviewer's round-one report, and the associate editor's private triage note.
The monitor works as follows. For a submission $x$, it executes the pipeline once with secret $s$ and caches every message. For a counterfactual $s'$, it re-executes only from the first agent whose input contains the secret. All upstream messages come from the shared prefix, following the prefix-tree idea of RVHyper. At each downstream boundary, it collects $k$ samples per secret value and runs a two-sample test. We use maximum mean discrepancy on sentence embeddings. We also run a classifier two-sample test, in which a small model is trained to tell the two groups of messages apart and its held-out accuracy is the statistic. Many boundaries and submissions are tested at once, so we control false discoveries across them with Benjamini–Hochberg. The monitor reports the earliest boundary at which the null hypothesis is rejected. That boundary localizes the leak to the agent just upstream of it.
The reason boundary-level checks could detect more than end-to-end checks is simple. The final decision is a single bit. A leak can change what crosses an internal boundary while leaving that bit unchanged on most submissions, or it can move the bit only rarely. Rich intermediate text should therefore expose the leak with fewer samples. Whether this holds in practice is the main empirical question.
We also test a compositional claim in the spirit of Hypercontracts. If every boundary passes its local test, does the end-to-end property also hold? This is not guaranteed. Statistical errors accumulate, and an agent can rebuild a secret from several inputs that each look harmless on their own. We measure how often local passes coincide with global failures.
Baselines
The naive reference point does nothing. It runs the pipeline with input redaction (author names, affiliations, and acknowledgments stripped) and assumes the silos hold. It measures how much leakage survives the standard engineering fix.
The strongest competitor is end-to-end counterfactual swap testing. This is the standard bias-audit practice: run the whole pipeline twice with the secret swapped and compare final decisions, plus final review text, with the same two-sample tests. No paper in the file reports a result for this audit on review pipelines, so there is no reported result found on Lacuna for end-to-end counterfactual auditing of LLM review pipelines. It is the strongest baseline because it is exactly what a practitioner would do, and it already is a two-run non-interference check, just with a single observation point. For fairness, it gets the same total budget of model calls as our monitor, the same submissions, the same samplers and temperatures, the same test statistics, and the same false-discovery control. Our monitor beats it if it reaches higher detection power on planted leaks at equal call budget, or the same power with substantially fewer calls. Both systems should hold the false-positive rate on leak-free pipelines at the nominal 5% level. We also report how often each correctly names the leaking agent.
A third baseline is the probing attacker. A classifier trained on intermediate messages tries to predict the secret, which tests whether any single message carries the secret. This is the strongest single-run alternative.
The ablation that isolates our key ingredient keeps boundary-level testing but removes prefix sharing, re-running the full pipeline for every counterfactual. A second ablation keeps prefix sharing but tests only the final output. Together they separate the contribution of where we observe from the contribution of how we save calls.
For pipeline utility, the reference is the fine-tuned Qwen2.5-7B-Instruct on FIRSTPASS outcome prediction, which reports 80.5% accuracy and 78.2% macro-F1 FIRSTPASS. We use it to check that repairing the leaks we find does not degrade decision accuracy.
Why not just…?
Why not just redact identifying fields before the pipeline runs? Redaction stops the obvious channel but leaves indirect ones. Writing style, self-citations, and dataset names can reveal authors. Agent-to-agent summaries can also reintroduce information that one agent saw legitimately. This is the same point the blockchain access-control papers face: Yang et al. control who reads which record, and that does not cover what is said afterward. Our planted-leak suite includes leaks that pass through redaction, and the do-nothing baseline measures directly how many survive.
Why not just swap the secret end to end and compare decisions? This is the real threat, and the proposal is built to be falsified by it. If the end-to-end swap, given the same call budget, matches our monitor's detection power and localization on planted leaks, then the boundary-level machinery adds nothing. In that case we would report the cheaper method. Our prediction that it will not match rests on two factors. A binary decision is a much weaker observation than internal text. And re-running only downstream of where the secret enters saves calls in proportion to how late that happens, a saving that RVHyper demonstrated in a deterministic setting.
Why not just train a probe to read the secret off each message? A probe tests whether a single run's text reveals the secret. That is a weaker question than whether downstream behavior depends on it. An agent can be steered by a cue without restating it, which leaves the probe nothing to find. We include the probe as a baseline and as one of the monitor's test statistics. If probe accuracy alone explains our detections, the paired formulation is unnecessary, and the experiment will show whether that is the case.
Proposed Experiments
The decisive experiment uses a planted-leak suite built on FIRSTPASS submissions across its five disciplines, with ResearchTown's ReviewBench tasks (research-bench) as a second source of review prompts. We create pipeline variants with known leaks of graded strength. An associate editor's summary includes an affiliation cue with probability $p$. A reviewer receives the other reviewer's score before writing. An editor's private triage note is appended to the decision letter. Each variant is paired with leak-free controls. For each method, we trace detection power and false-positive rate as functions of the model-call budget, and we record localization accuracy. Our claim survives only if boundary monitoring dominates the end-to-end swap at small $p$ and stays calibrated on the controls.
The second experiment audits unmodified pipelines built with off-the-shelf models, with no planted leaks. We report which boundaries fail for author identity and for reviewer independence. We then apply targeted repairs: restricting a single prompt for the failing agent, or blocking one message field. We re-check FIRSTPASS outcome accuracy against the 80.5% reference to make sure the repairs cost little.
The third experiment tests composition. Across all audited pipelines, we count cases where every boundary passed but the end-to-end test failed. A high count would mean local checks cannot replace a global audit and must be supplemented with it. That would be a useful negative result about carrying hypercontract-style reasoning over to statistical agents.
Finally, we vary the topology between flat teams and editor-led hierarchies, following Guo et al., and we vary the number of subfield tracks. This tests whether hierarchical structure creates more leak points or fewer, given that hierarchies are adopted for efficiency.
References
- Deciding Hyperproperties
- Efficient monitoring of hyperproperties using prefix trees
- Probabilistic Hyperproperties with Nondeterminism
- Parameter Synthesis for Probabilistic Hyperproperties
- Hypercontracts
- HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- ResearchTown: Simulator of Human Research Community
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
- Permissioned blockchain technologies for academic publishing
- Blockchain-based Enterprise Community Data Access Control Methods and Applications
- Group Fairness in Peer Review