Mechanism-Routed Review of Permanence Claims in Contaminant Immobilization

Papers on cleaning up toxic metals in soil and water often say a material "immobilizes" cadmium, lead, arsenic or chromium. The evidence behind that word is frequently a one- to two-day lab test in which the metal sticks to the material. That shows the metal binds quickly. It does not show the metal stays bound for decades as rain acidifies the soil, oxygen returns to a waterlogged layer, competing metals arrive, or the material itself decays. The field's own reviews make this point. Adsorption of Pb2+ by ameliorated alum plasma in water and soil and the mining-site preface Preface: Heavy metal(loid)s at mining & metallurgical sites: Fate, risk and remediation both note that short batch tests are routinely read as evidence of permanent stabilization. We propose an automated review system, modeled on how a journal's editorial board is structured, that reads a remediation paper and judges whether its permanence claim is backed by tests of the specific ways that particular binding mechanism is known to fail.

The key observation is that the failure modes depend on the mechanism, not on the paper in general. Metal held in sulfide minerals made by bacteria (Immobilization Behavior and Mechanism of Cd2+ by Sulfate-Reducing Bacteria in Anoxic Environments) is threatened by re-oxidation. Arsenic on schwertmannite is threatened by the mineral transforming into another phase (Adsorption characteristics of As(III) by schwertmannite: new findings in mineral-phase transformation and microbial effects). Cadmium on biochar is threatened by competing zinc and by soil alkalinity shifts (Competitive Adsorption of Cd(II) and Zn(II) on Biochar, Loess, and Biochar-loess Mixture), and by the biochar itself oxidizing over years. A generic "did they run a long test?" check misses a paper that ran a 60-day aging test but never re-exposed its sulfide product to air.

Core Claim

A two-tier review, structured like an editor-in-chief with associate editors, should catch mechanism-specific permanence overclaims as well as or better than a full multi-agent reviewer, at a small fraction of the cost. In this design, a cheap first tier routes most papers. Only papers whose claims are contested go up to the second tier, where expert agents write down in advance how that mechanism should be stress-tested.

The second tier reuses the central design choice of MARG: Multi-Agent Review Generation for Scientific Papers. MARG uses expert agents that draft quality criteria without seeing the manuscript, so the paper cannot anchor what they expect. A leader agent then questions worker agents that hold chunks of the text. Our system differs from MARG in two ways:

  • Mechanism routing. The criteria are produced per immobilization mechanism, not per generic review aspect.
  • Escalation. The expensive multi-agent stage is invoked only on demand. MARG-S runs it on every paper, consuming over 1.2 million input tokens and about an hour per review.

We treat the escalation tier as a hypothesis, not a given. If the first tier alone matches it, that becomes the main result.

Research Program

Benchmark. The first deliverable is a benchmark for permanence claims, since none exists. Its construction follows Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers, which built LimitGen-Syn by inserting known limitations into arXiv papers and LimitGen-Human from real ICLR limitations. That work is restricted to AI papers; we transfer its construction to environmental remediation.

The synthetic subset starts from remediation papers that do report durability evidence: leaching under acidic conditions, wet–dry or redox cycling, competing-ion tests, aging of the amendment, or post-test mineral phase identification. We then make controlled edits that remove or weaken one such test while leaving the permanence claim intact. Each edit is labeled with the mechanism and the missing stressor. The source corpus spans the material classes in the file:

We deliberately include "decoy" items, where a generic long-term test is present but the mechanism-specific one is absent.

The human subset comes from the Chemistry and Earth Science records of FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes. That dataset holds 3,668 multi-round Nature Communications reviews linked to real editorial outcomes. Domain annotators will tag every reviewer comment that questions durability, re-release, or long-horizon stability, and record which mechanism it targets. FIRSTPASS has no such taxonomy, so this labeling is new.

Tier 1 (desk screen). A single model extracts three things: each permanence claim, the binding mechanism the paper asserts, and the tests reported. It then compares these against a fixed table that maps each mechanism to its known stressors:

  • surface adsorption and ion exchange → pH and competing ions;
  • surface precipitation → acidification;
  • biogenic sulfide → re-oxidation;
  • metastable minerals → phase transformation and microbial activity;
  • organic matrices → oxidative aging;
  • cement matrices → weathering;
  • spent adsorbents → disposal leaching.

We phrase every table entry positively ("the paper should report re-oxidation leaching"), not as a prohibition. This follows Evaluating LLM Agent Adherence to Hierarchical Principles: A Lightweight Benchmark for Verifying AI Safety Plan Components, which found near-perfect adherence to a positively framed principle and high variance on negatively framed ones.

Escalation and Tier 2. Tier 1 outputs a confidence score. It escalates a paper when that confidence is low, when the asserted mechanism is ambiguous or mixed, or when the paper claims a mechanism absent from the table. In Tier 2, one expert agent per candidate mechanism writes its stress-test expectations blind to the text. A leader agent then interrogates workers holding chunks of the paper, and a fact-checking pass removes critiques the full text refutes, as in MARG's refinement stage. The output is a structured verdict: supported, partially supported, or unsupported, together with the missing stressor.

Secondary use. We also test transfer of the trained table to carbon-storage site assessments such as 3D seismic modeling of the Amal oil field to evaluate CO2 storage potential in depleted reservoirs, Southern Gulf of Suez. That paper argues containment depends on slow geochemical interaction with host rock. We expect this transfer to be weak and report it only as a probe.

Baselines

Naive reference. A single frontier model is given only the instruction to judge whether long-term stability is adequately supported. This is the "do nothing" reference. On the closest existing task, standalone GPT-4o reaches 52.0% coarse accuracy on LimitGen-Syn (Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers). The same paper reports a 12.2% gain from adding literature retrieval through Semantic Scholar, so we also run a retrieval-augmented single-agent variant.

Strongest existing method. MARG run unmodified, every paper, full pipeline. It is the strongest because it is the best reported system for this kind of flaw detection: 68.1% coarse accuracy on LimitGen-Syn, against 52.0% for GPT-4o. In the separate user study it produced 3.7 good comments per paper versus 1.7 for single-agent GPT-4 (MARG: Multi-Agent Review Generation for Scientific Papers). These figures are on AI papers. There is no reported result on Lacuna for any system on remediation permanence claims.

Supervised outcome baseline. On the FIRSTPASS subset, a fine-tuned Qwen2.5-7B-Instruct classifier. This model reaches 80.5% accuracy and 78.2% macro-F1 at predicting standard (two-round) versus extended (three or more rounds) review, 10.4 points above zero-shot Gemini (FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes). We retrain it to predict our durability-concern labels instead.

Fairness. All LLM systems use the same backbone model, the same paper text, and the same retrieval access. Prompt tuning gets a matched budget on a held-out development split. Token use and wall-clock time are reported for every run.

What counts as beating MARG. Our system must match or exceed MARG's detection accuracy on the full synthetic set within a 95% bootstrap interval, while using at most a quarter of its tokens. It must also exceed MARG on the decoy items.

Ablations. These isolate the key ingredients:

  • Tier 1 alone, which is the static mechanism checklist.
  • A generic-durability checklist with no mechanism column.
  • Tier 2 with experts allowed to see the paper, which removes the blind-drafting decoupling.
  • A single generic "durability expert" in place of mechanism-routed experts.
  • Always-escalate, which isolates the cost saving.

Why not just…?

Why not just use one strong model with a detailed geochemistry prompt? The available evidence says single agents produce generic critiques: 60% generic comments for single-agent GPT-4 versus 29% for MARG-S. It also says they detect fewer inserted flaws (52.0% vs 68.1%). That evidence comes from AI papers, however. A long, mechanism-aware prompt might close the gap in a narrow domain. The naive and retrieval-augmented baselines, given the same mechanism table inside the prompt, test exactly this.

Why not just enforce a static checklist? This is the real threat, and we build it in as Tier 1 rather than argue against it. The research file's case against static checklists is that environmental systems change over time and rigid rules misfire, which is weak evidence. The honest test is the decoy items. There, a generic durability test is present but the mechanism-specific one is missing, or the mechanism is mixed, as with biochar combined with fungi or competing metals. If the mechanism-indexed checklist matches Tier 2 on those items too, we will report that the useful contribution is the table plus the benchmark, and that the hierarchy is unnecessary.

Why not just run full MARG on everything? It is the strongest detector, but its cost of more than 1.2 million tokens per paper makes routine screening of a journal's submission stream impractical. Running MARG on every paper is included as the always-escalate ablation. The question is whether routing loses accuracy, and our comparison measures that directly.

Why not just predict editorial outcomes with a fine-tuned classifier? The FIRSTPASS classifier is accurate at predicting review length. But it outputs a label, not the missing stressor, so it cannot tell an author what test to run. We compare it on durability-concern detection and expect it to be competitive on the yes/no flag and unable to localize the gap.

Proposed Experiments

Experiment 1: the main comparison. Every system is scored on the synthetic subset. Coarse accuracy asks whether the system flagged the missing durability evidence. Fine accuracy asks whether it named the correct mechanism-specific stressor. Results are reported separately for decoy and non-decoy items, alongside tokens and minutes per paper.

Experiment 2: real reviewer concerns. On the FIRSTPASS Chemistry and Earth Science subset, we measure recall and precision against human-reviewer durability comments. We then ask whether flagged papers are more likely to have gone to extended review, using FIRSTPASS outcomes as an external check that our flags track concerns real editors acted on.

Experiment 3: expert evaluation. Environmental chemists will blindly rate generated critiques for specificity and correctness, following MARG's user-study protocol.

The decisive check is the decoy subset:

  • If the mechanism-agnostic checklist and the single generic expert fail there, while mechanism routing succeeds, the core claim holds.
  • If Tier 1 alone succeeds there, the hierarchy is unnecessary.
  • If nothing succeeds, the stressor table is incomplete, and the failure cases tell us which mechanisms need it extended.

References

  1. Adsorption of Pb2+ by ameliorated alum plasma in water and soil
  2. Preface: Heavy metal(loid)s at mining & metallurgical sites: Fate, risk and remediation
  3. Immobilization Behavior and Mechanism of Cd2+ by Sulfate-Reducing Bacteria in Anoxic Environments
  4. Adsorption characteristics of As(III) by schwertmannite: new findings in mineral-phase transformation and microbial effects
  5. Competitive Adsorption of Cd(II) and Zn(II) on Biochar, Loess, and Biochar-loess Mixture
  6. MARG: Multi-Agent Review Generation for Scientific Papers
  7. Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers
  8. Immobilization of Cd(II) by phosphorus-modified bamboo biochar from solution: mechanistic study from qualitative to quantitative analysis
  9. A novel biochar-based 3D composite for ultrafast and selective Cr(VI) removal in electroplating wastewater
  10. Synergistic application of canna-derived biochar and arbuscular mycorrhizal fungi enhances cadmium immobilization and soil ecological function in contaminated soils
  11. Mechanistic study of the effect of potassium ferrate and straw fiber on the enhancement of strength in cement-based solidified municipal sludge
  12. FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
  13. Evaluating LLM Agent Adherence to Hierarchical Principles: A Lightweight Benchmark for Verifying AI Safety Plan Components
  14. 3D seismic modeling of the Amal oil field to evaluate CO2 storage potential in depleted reservoirs, Southern Gulf of Suez