Observability-Aware Soft Suppression for Multi-Agent Clinical Extraction

A hospital record for one patient is usually too long and too scattered for one model to read at once. The practical fix is to split it: several language-model "agents" each read a slice, such as the pathology reports, the nursing notes, or one week of progress notes, and each proposes structured facts like "metformin, 500 mg, active." Every such system then needs a merge step that decides which proposed facts are duplicates, which are distinct, and which contradict each other. Today that step is crude. Systems drop conflicting features, rerun prompts, or follow fixed routing rules. Each agent saw only part of the record, so two candidates that look alike may be one fact seen twice, two different events (a past and a current prescription), or a real contradiction that exists only because one agent never saw the evidence the other relied on.

We propose to treat this merge step as a suppression problem, borrowing from object detection. In detection, Soft-NMS (soft non-maximum suppression) lowers the score of an overlapping candidate box instead of deleting it. Our version adds one ingredient that comes from the multi-agent setting. A higher-scoring candidate from agent $j$ may suppress a candidate from agent $i$ only to the extent that $j$ could actually have seen $i$'s evidence. When $j$ could not have seen it, the overlap is treated as unexplained. The candidate is kept, and the merge sends a single targeted evidence snippet to $j$ to settle it. The result is a cheap, deterministic, auditable merge rule that produces calibrated scores. Our claim is that it keeps rare and dispersed facts which current merging throws away.

Core Claim

In multi-agent extraction over split clinical records, the main merge error is suppression without observation: a confident agent's extraction overrides a peer's extraction about evidence it never read. A suppression rule that combines overlap in meaning with overlap in what each agent could see should recover facts that hard merging loses. The gains should concentrate in three places: rare concepts, indirect relations whose support sits in another section, and settings where each agent's view is small. The rule should also lose less accuracy than other merges as the record is cut into more pieces.

The Mechanism

Each agent $a$ reads a view $V_a$, a known set of documents, sections, and time ranges. It emits candidates $c_i$, each carrying a normalized concept, attributes (dose, status, timing, negation), a verbatim evidence span, and a confidence score $s_i$. Candidates are processed in score order, following Soft-NMS. That paper replaced hard deletion of overlapping boxes with Gaussian score decay and gained 1.1 to 1.7 mAP points without retraining. When a kept candidate $c_j$ is compared against a lower-scoring candidate $c_i$, we update

$$s_i \leftarrow s_i \cdot \exp\!\left(-\frac{(o_{ij}\,\kappa_{ji})^2}{\sigma}\right),$$

where $o_{ij} \in [0,1]$ is the overlap in meaning between the two candidates (concept match times agreement on attributes), $\kappa_{ji} \in [0,1]$ is the estimated chance that agent $j$'s view contained the evidence behind $c_i$, and $\sigma$ sets how sharply scores decay. With $\kappa_{ji}=1$, every agent sees everything and the rule reduces to ordinary Gaussian Soft-NMS over concepts. With $\kappa_{ji}=0$, the two candidates come from views that do not intersect, so neither can suppress the other.

Coverage $\kappa$ is estimated from metadata and simple retrieval. We check whether $c_i$'s evidence document, section, or date range falls inside $V_j$, and whether its evidence span or a near paraphrase appears in $V_j$. One case is handled separately. When $o_{ij}$ is high on the concept but the status attributes conflict (affirmed versus negated, active versus discontinued) and $\kappa_{ji}$ is low, the merge does not decay either score. It sends $c_i$'s evidence span to agent $j$ and asks for a single re-judgment. Communication is therefore triggered by suppression conflicts and limited to one snippet per conflict.

This differs from the single-model span work that brought Soft-NMS into text. Locate and Label generates candidate spans with boundary regression and decodes them with Soft-NMS to keep nested entities, gaining +3.08 F1 on KBP17. ESD uses Beam Soft-NMS for few-shot span labeling. Both papers supply our method, but in both the overlap is token IoU inside one passage that a single model saw in full. Here the overlap is in concept space, across agents with different views.

Positioning Against Prior Work

The clinical multi-agent systems motivate the proposal, and their merge steps are what we replace. SNOW splits feature generation into proposal, alignment, extraction, validation, and code-generation agents. It matched manual abstraction (AUC 0.767 vs 0.762), but it drops or reruns conflicting features and assumes full note access at validation. nMAS routes oncology attributes across rule-based, small-model, and large-model tiers. A multiplexer with static rules combines their outputs, and the system reaches 85.0% rank-weighted F1 against 66.4% for a single universal model. That gap tells us decomposition matters. It does not tell us how to merge what the pieces return.

Self-Verification is the strongest single-model competitor. It runs extraction, omission detection, evidence grounding, and pruning as one chain. Its pruning is a hard keep-or-discard decision, and one context reviews its own output.

The partial-observability literature shapes our design. MACRO-LLM shows that language-model agents with local views can coordinate through compressed, constant-size messages, in traffic and epidemic control. We take the lesson that messages should be small and triggered by need, and we apply it to text. Reference-Centric Models for Grounded Collaborative Dialogue re-ranks utterances by simulating what the partner can see. Our $\kappa$ is a deliberately crude, explicit version of that perspective-taking.

We do not adopt learned credit assignment of the COMA or QMIX kind. Those methods need reward signals over many episodes, and labeled clinical records are too scarce to supply them. Choosing a fixed, inspectable rule is a scoping decision, not an oversight.

Research Program

The first component is a controlled partial-observability harness. Each record is split into $K$ views along realistic boundaries: section type, using MedSecId-style headers from MedSecId, or time windows. Each view goes to an agent running the same extraction prompt and model. Varying $K$ from 1 (full view) to many small views gives a dial on observability while holding the extractor fixed.

The second component is the merge rule above, plus the conflict-triggered snippet query. Both are implemented as a model-agnostic post-processor. The parameter $\sigma$ and the conflict threshold are tuned on development splits.

The third component is an error analysis that sorts every merge decision into four categories: correct duplicate removal, wrongful suppression of a distinct fact, wrongful retention of a duplicate, and contradiction resolution. The goal is to test directly whether wrongful suppressions have low $\kappa$ under the baseline merges, which is the mechanism the claim predicts.

Baselines

The naive reference point is union with exact-match deduplication. Every agent's candidates are kept, and only identical normalized tuples are collapsed. This sets the recall ceiling and the precision floor of the agent outputs.

The second baseline is hard merging, the analogue of greedy NMS. It keeps the highest-confidence candidate for each concept and discards overlapping lower-scoring ones. This approximates the drop-on-conflict behavior of SNOW and nMAS-style multiplexing, implemented ourselves because neither system is released for these benchmarks.

The key ablation is plain concept-level Gaussian Soft-NMS with $\kappa$ fixed at 1, in the spirit of Locate and Label. It isolates whether the observability term adds anything beyond soft decay. A second ablation removes the snippet query, to separate the decay rule from the communication step.

The strongest existing method is Self-Verification. On MIMIC-IV discharge-summary ICD-10 extraction it reports F1 0.533, up from 0.487 for a single pass. We run it with the same base model, both on full documents where they fit and chunked otherwise.

We also run an LLM-judge merge. A separate agent sees all candidates with their evidence spans and adjudicates. No reported result was found on Lacuna for this configuration on these benchmarks.

As further references we include a single long-context pass over the whole record and a full-observability multi-agent run in which every agent reads everything. The latter is an upper reference for what the merge could achieve if views were not split.

All methods use the same base model, the same splits, and an equal number of development-set tuning runs. Each method is also reported at matched total input-token budgets. Beating the strongest competitors means three things together. On the full test sets, the method must exceed Self-Verification and the LLM judge in F1 at equal or lower token cost. On the rare-concept and indirect-relation subsets, it must show a statistically significant recall gain with no precision loss. As $K$ grows, its accuracy must fall more slowly than every other merge.

Why not just…?

Why not just give one long-context model the whole record? The file's evidence leans against it, though we treat it as a live threat. On BioDEX, few-shot GPT-4 reaches 53.1% F1 on document-level adverse-event extraction against 72.0% for humans, and the authors identify document length as a binding constraint. In nMAS, the universal single model trails the decomposed system, 66.4 against 85.0. Those comparisons use older models or other domains. We therefore include the $K=1$ condition and break results down by record length. If the long-context pass matches us on the longest records at similar cost, the method is useful only where records exceed the context limit or sit in separate systems, and we will report that limit.

Why not just let an LLM judge reconcile the candidates? This is the most serious alternative, and it may match us on accuracy. A judge given every candidate with its evidence can reason about duplicates and contradictions in context. It also faces the same blind spot: it sees evidence snippets, not the views that produced them, and it can be biased toward confident, frequent claims. It also yields one decision per candidate rather than a ranked score, so no precision-recall curve can be traced. We compare at matched tokens and again at the judge's preferred budget. If the judge wins on accuracy, the remaining contribution is cost and auditability. We would then test whether feeding $\kappa$ to the judge as an input closes the gap, which would relocate the idea rather than abandon it.

Why not just use ordinary Soft-NMS over concepts? If decay alone explains the gains, the observability term is unnecessary and the claim collapses into a straightforward port of Locate and Label to agent outputs. The $\kappa=1$ ablation settles this. The specific prediction is that the two rules agree at $K=1$ and diverge as views shrink, with the plain version wrongly suppressing more low-coverage candidates.

Why not just run self-verification inside each agent? Per-agent verification catches omissions within a view. It cannot detect that a fact one agent kept as "active" is "discontinued" in another agent's view. We combine it with our merge as well as running it standalone, since the two are complementary rather than exclusive.

Proposed Experiments

The primary test uses MIMIC-IV discharge summaries for ICD-10 extraction, the setting of Self-Verification. Summaries are split by section and by length into $K \in \{1, 2, 4, 8\}$ views, and we report micro-F1 against the 0.533 reference.

The test of the cross-section claim is the 2022 N2C2 Track 3 Assessment and Plan task. Agents read the Subjective, Objective, and Assessment-and-Plan sections separately, and the merged evidence feeds the relation label. The reported Indirect-class F1 of 0.6713, against a best overall macro F1 of 0.8212, is exactly where support lives in sections a single reader may lack. We predict the largest gains on the Indirect class.

For overlapping and nested medication facts, we use the 2018 n2c2 Track 2 discharge summaries. MC-DRE reports 98.38% F1 there under full observation. Because that is near ceiling, we measure how much each merge loses from that full-view level as $K$ grows, with a focus on adverse-event and reason entities.

The rarity check uses the 2019 n2c2 normalization corpus from Robust Benchmarking. Accuracy there falls from 85.2% on all spans to 7.0% on unpopular concepts. We test whether hard merging widens that gap and whether observability-aware decay narrows it. BioDEX serves as an out-of-domain document-level check against the reported 53.1 (few-shot GPT-4) and 59.1 (fine-tuned FLAN-T5-Large) F1.

Three results would decide the question. The plain-Soft-NMS ablation must fall behind the full rule as $K$ increases. The rule must gain recall on the unpopular-concept and Indirect subsets without losing precision. In the error analysis, most wrongful suppressions made by hard merging must be pairs with low $\kappa$. If the first of these fails, we drop the observability claim and report concept-level Soft-NMS as the useful finding.