Neutrophil extracellular traps (NETs) are web-like meshes of DNA and toxic granule proteins that immune cells extrude to ensnare pathogens; when released in excess, they shred tissue, clog microvasculature, and — as recent work shows — even dissolve the gut's lymphoid organs after a stroke. The dominant therapy is enzymatic: DNase I cleaves the DNA scaffold and collapses the web. But the literature documents a trap inside the trap. DNase I dissolves the lattice while leaving citrullinated histones and active neutrophil elastase (NE) untouched, potentially liberating soluble cytotoxic proteases into circulation (Neutrophil extracellular traps are indirectly triggered by lipopolysaccharide and contribute to acute lung injury). Correct NET therapy is therefore multi-step reasoning — pick the target, pick the route, anticipate what the degradation itself releases, monitor distant immune compartments — and every step can silently poison the ones downstream.
LLM-as-a-judge — prompting a large language model with an explicit rubric to grade open-ended outputs — has reached near-physician reliability on clinical text (an intraclass correlation of 0.818 against seven physician raters, in 22 seconds instead of 600, Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge) — but it collapses on mechanistic immunology: even the strongest models reject only about 20% of expert-written false claims in vaccine-adjuvant research (An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM). This proposal builds the first LLM-judge benchmark for NET degradation therapy, and tests one specific claim: the decisive failures in NET therapeutic reasoning are propagated mechanistic omissions that flat, end-output grading cannot see, but that step-level evaluation — scoring each node of a therapeutic decision chain against its upstream context — can.
Core Claim
A judge that decomposes a NET therapeutic plan into a directed acyclic graph (DAG) of decision nodes, scores each node against a formal representation of NET mechanism chains, and propagates failures back to their root cause will substantially outperform the strongest flat rubric-based judge at detecting mechanistic errors — such as "DNase I monotherapy is sufficient" or "post-stroke patients only need lung monitoring" — while matching flat judges on overall expert agreement. The motivation is direct: on agentic software workflows, flat end-to-end evaluation caught only 41% of failures while DAG-structured step-level evaluation caught 89%, a 2.17× gap in failure-detection recall (AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking). No one has tested whether that gain transfers to biomedical therapeutic reasoning, where the errors are semantic rather than syntactic. The honest possibility that it does not is itself a decisive outcome of this work.
Research Program
We will build NET-Bench, a three-part artifact grounded in the NET corpus. First, a mechanism representation: following the adjuvant benchmark's method of translating fuzzy biological text into computable function-and-variable descriptions (An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM), we encode the NET mechanism space from the five NET papers — triggers (PMA, LPS acting indirectly through activated platelets Neutrophil extracellular traps are indirectly triggered by lipopolysaccharide and contribute to acute lung injury, acid aspiration Neutrophil extracellular traps contribute to the pathogenesis of acid-aspiration-induced ALI/ARDS, high-tidal-volume ventilation partially dependent on TLR4 and HMGB1 Neutrophil Extracellular Traps Are Pathogenic in Ventilator-Induced Lung Injury and Partially Dependent on TLR4, pristane-induced lupus hemorrhage Necrostatin-1 Alleviates Diffuse Pulmonary Haemorrhage by Preventing the Release of NETs via Inhibiting NE/GSDMD Activation in Murine Lupus), execution machinery (NE, GSDMD cleavage, PAD4), effectors (the DNA scaffold, Cit-H3, NE, MPO; NETs at ≥500 ng/mL impair epithelial proliferation), systemic consequences (Peyer's-patch melting and IgA loss after stroke or myocardial infarction Stroke and myocardial infarction induce neutrophil extracellular trap release disrupting lymphoid organ structure and immunoglobulin secretion), and interventions with their documented blind spots (intratracheal DNase I, the NE inhibitor alvelestat, Nec-1, Cl-amidine).
Second, evaluation items: roughly 200 therapeutic plans for three scenarios (an acid-aspiration ARDS patient, a lupus patient with diffuse alveolar hemorrhage, a post-stroke patient at risk of secondary pneumonia), each either expert-written or LLM-generated, with a subset seeded with one planted mechanistic error at a randomized depth of the decision chain. The errors are drawn from the literature's own documented tensions: DNase I recommended without an elastase inhibitor; systemic IgA collapse and secondary-infection risk ignored because the primary insult was cerebral; an experimental claim of "direct LPS induction of NETosis" accepted despite the platelet-contamination artifact; N-GSDMD activation asserted in vivo despite the acknowledged lack of a validated antibody. Third, the judge: a GPT-4o-class model scores each DAG node (target selection, delivery route, dosing timing, downstream immune monitoring) against the formal representation and its upstream context, flags failures, and attributes root causes, reusing AgentEval's architecture but with a biomedical rubric. A multimodal extension using confocal co-localization figures will follow the strict filtering discipline of VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering, which kept only 3.0% of candidate pairs; the transcriptomic and proteomic evidence behind the stroke findings is publicly accessible under NCBI GEO GSE254410 and ProteomeXchange PXD044644 and will anchor evidence-node questions.
Proposed Experiments
Experiment 1 measures judge reliability: five to seven NET domain experts (mirroring the seven-physician panel design of the healthcare-judge study) independently grade a held-out set of 100 plans; we report intraclass correlation and Pearson correlation per rubric dimension. The bar is the adjuvant benchmark's DeepSeek-R1 human correlation of 0.9125 (An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM), achieved in an easier immunology subdomain — we treat ≥0.85 on a harder domain as success.
Experiment 2 is the decisive test: 200 single-error plans, flat judge versus DAG judge on failure-detection recall, root-cause attribution accuracy, and Cohen's kappa against expert-labeled error locations. AgentEval's flat 0.41 versus step-level 0.89 on software workflows motivates the hypothesis; whether a 2× style gap survives when errors are mechanistic omissions rather than code defects is exactly what this experiment settles. Experiment 3 measures hallucination rejection on roughly 50 expert-written false mechanistic claims about NET therapy, reported as a Hallucination Rejection Ratio against the adjuvant benchmark's median of only ~20% for top models. A calibration control guards against the judge becoming a DNase-alarmist: correct plans that legitimately combine DNase I with alvelestat (as validated in the acid-aspiration study) must not be flagged. Experiment 4, the multimodal phase, asks whether OCR-decoupled text pipelines paired with extracted image features beat native vision-language judging on confocal NET imagery, a deficit already documented for dense scientific microscopy.
Baselines
The naive reference point is lexical-overlap scoring — grading a plan by BLEU and ROUGE against an expert reference answer, run on identical items with negligible compute. It is included because it is the cheap default, not because it is strong: in the medical-QA study, BLEU correlated with expert clinical ratings at only r = 0.65, and GPT-5-mini scored a BLEU-1 of 0.0124 while producing clinically meaningful answers (Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation). Beating it means materially higher expert correlation and any non-trivial failure-detection recall, since overlap scoring cannot localize a mechanistic omission at all.
The strongest existing competitor is the flat monolithic rubric judge: a single reasoning model, few-shot prompted with an explicit domain rubric — the configuration that achieved ICC 0.818 against physicians in clinical summarization (Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge) and Pearson 0.9125 with immunology experts in the adjuvant framework (An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM). To make the comparison fair, the flat judge uses the same underlying model, the same rubric text and domain facts, and the same total inference budget as the DAG judge — only the decomposition differs. We beat it only if failure-detection recall and hallucination rejection are materially higher at no loss in expert agreement. An intermediate competitor is the checklist-milestone approach of ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems, which grades against expert-defined logical milestones without propagation; and an off-the-shelf reward model, with ArmoRM-Llama3-8B (overall 89.0 on RewardBench) as representative, run with the same items. The key ablation runs the full system minus one ingredient at a time: no DAG decomposition (single prompt), and no formal representation (natural-language facts only), isolating which component carries the gain. If the flat judge with our domain rubric matches the DAG judge, the honest conclusion is that the benchmark and its error taxonomy — not the judging architecture — is the contribution, and we will report it that way.
Why not just…?
Why not just score against expert answers with BLEU or ROUGE? The medical-QA study directly quantified this: BLEU's correlation with expert clinical ratings was moderate (r = 0.65), models with low overlap were often clinically superior, and lexical metrics cannot detect safety violations or subtle factual hallucinations (Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation). Our planted errors — an omitted elastase inhibitor, an unmonitored gut — are precisely the class that leaves surface text fluent and complete. One caveat we take seriously: ROUGE-1's correlation of r = 0.89 with judge scores in that same study means lexical metrics are deceptively strong on global agreement; this is why our primary endpoints are failure-detection recall and hallucination rejection, not overall correlation, and Experiment 2 includes the lexical baseline on those endpoints.
Why not just have human immunologists review everything? Humans are the gold standard and remain so here — they author our ground truth. But the healthcare-judge study measured the cost: seven physicians averaged 600 seconds per clinical summary while the LLM judge took 22 seconds, a 96% reduction, at ICC 0.818 (Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge). NET therapy spans a sprawling preclinical corpus; expert review cannot scale to it. Our design keeps experts in the loop where they matter and tests where the judge disagrees with them.
Why not just prompt one strong LLM with a good rubric? This is the real threat, and we do not argue around it. Evidence says flat judging misses propagated intermediate failures — recall of 0.41 versus 0.89 for step-level DAG evaluation on workflows (AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking) — and flat judging accepts the majority of expert-written false immunology claims (median hallucination rejection ~20%, An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM). But those results are from other domains; the flat-judge-with-our-checklist ablation in Experiment 2 is exactly the experiment that settles whether decomposition adds anything once the domain facts are in the prompt, and we commit to reporting a negative result if the gain evaporates.
Why not just use an off-the-shelf reward model? RewardBench documents two failure modes that are near-fatal for this domain: pervasive length bias, and the refusal trap, where reward models reflexively reward refusals whenever cytotoxic or destructive mechanisms are mentioned (RewardBench: Evaluating Reward Models for Language Modeling). NET literature is saturated with exactly such vocabulary — tissue destruction, cell lysis, cytotoxicity — so a reward model would systematically punish correct therapeutic discussion; and RewardBench found reasoning showed the widest variation across the more than 80 models tested. We still run ArmoRM as a baseline in Experiments 2 and 3 so the claim is measured rather than asserted.
Decisive Checks
The proposal stands or falls on three numbers. The DAG judge must beat the flat judge on failure-detection recall for planted mechanistic errors by a margin large enough to matter clinically (a pre-registered threshold of at least 15 percentage points, in the spirit of the 41-to-89 gap that motivates the design). Expert agreement must reach at least ICC 0.85 on the held-out set, within sight of the 0.9125 human correlation the adjuvant benchmark achieved in an easier domain. And hallucination rejection on NET-specific trap claims must clear roughly 50%, well above the ~20% median the adjuvant study reported, while the calibration control confirms the judge still approves legitimate DNase-plus-inhibitor combinations. If all three hold, the deliverable is a reusable benchmark, a mechanistic error taxonomy for NET therapy, and evidence that step-level judging is what automated evaluation of therapeutic reasoning was missing.
References
- Neutrophil extracellular traps are indirectly triggered by lipopolysaccharide and contribute to acute lung injury
- Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge
- An Open-Ended Benchmark and Formal Framework for Adjuvant Research with MLLM
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Neutrophil extracellular traps contribute to the pathogenesis of acid-aspiration-induced ALI/ARDS
- Neutrophil Extracellular Traps Are Pathogenic in Ventilator-Induced Lung Injury and Partially Dependent on TLR4
- Necrostatin-1 Alleviates Diffuse Pulmonary Haemorrhage by Preventing the Release of NETs via Inhibiting NE/GSDMD Activation in Murine Lupus
- Stroke and myocardial infarction induce neutrophil extracellular trap release disrupting lymphoid organ structure and immunoglobulin secretion
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
- ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems
- RewardBench: Evaluating Reward Models for Language Modeling