Type-Reachability Pruning for Multimodal Agent Teams

When a team of language-model agents answers a question that spans text, tables, and images, most of the talking is wasted. The image agent describes objects nobody asked about. The table agent answers a question the text agent could not use. Every agent hears everyone else. This proposal gives the team a small ontology, meaning a typed map of the kinds of things in the domain and the relations that can hold between them. Each agent may speak only in typed statements drawn from that map. Before each round, a cheap symbolic check asks whether an agent's message could, even in principle, lie on a chain of relations that ends at an answer of the type the question demands. Agents and communication links that fail the check are skipped before they run.

The same typed vocabulary also serves as the shared language between the image agent and the text agents. That is how the proposal attacks the mismatch between dense visual input and sparse language. An image contains dozens of detectable things, but only those whose type can reach the answer are ever reported. In two sentences: learned pruning of agent communication reduces cost but cannot tell a useless message from a useful one by meaning, while ontology-guided agent teams reason well but run every agent every time. We use reachability in the ontology's type graph, recomputed after every round, to decide which agent-to-agent links can still matter. We test whether this beats both learned pruning and one-shot routing on multi-hop cross-modal question answering.

Core Claim

The claim is that dynamic, type-level reachability is a safe and cheap pruning signal for multi-agent multimodal reasoning. It stays safe as long as entity typing is correct, much as a violated anti-monotone constraint safely eliminates a subtree in constrained pattern mining (Pei et al., 2006). The payoff should be concentrated on questions where the right next agent depends on an intermediate result. A typical case is a question whose first hop, a table lookup, reveals a bridge entity whose type determines whether the second hop must look at an image or a passage. A router that decides once, at the start, cannot exploit that intermediate information. A learned edge-weight pruner can exploit it only statistically, over many rounds of training.

Let the ontology be a directed graph over classes, with edges labelled by relations. Let $S_t$ be the set of classes of entities already established after round $t$, $c^\*$ the class required of the answer, and $R_a$ the set of relations agent $a$ can resolve with its tools. The link from the current state to agent $a$ is kept at round $t+1$ only if some class in $S_t$ reaches $c^\*$ by a path in the class graph whose first edge uses a relation in $R_a$:

$$\text{keep}(a, t) = \exists\, c \in S_t,\ r \in R_a:\ c \xrightarrow{r} c' \leadsto c^\*.$$

This check costs a graph search over a few dozen classes, which is negligible next to one model call.

Relation to Prior Work

Three papers motivate the gap and one supplies the method. M3Prune learns sparse communication graphs for multimodal multi-agent retrieval. It relaxes edges with Gumbel-Softmax and adds a rank penalty, first within modality groups and then across them. It reports a 9.4% accuracy gain and a 15.7% token reduction across benchmarks that include MultimodalQA. Its edges carry no semantic types, so it cannot cut a link because a message is of the wrong kind; we replace learned edge weights with a symbolic test that needs no training. Adaptive Graph Pruning for Multi-Agent Communication likewise sparsifies agent networks without any ontology.

OG-MAR guides five persona agents and a judge with a cultural ontology. It improves average accuracy from 0.5602 to 0.6308 on Gemini 2.5 Flash Lite, but it runs every agent on every query, and its authors name agent pruning as future work. We supply that pruning and move it to a multimodal setting.

OaK supplies the method. It has a language model draft a schema, checks the schema with a formal reasoner, and restricts a single agent to typed functions, reaching a 55.90% final pass rate on TravelPlanner against 51.50% for the MemP baseline. We borrow its verified-schema construction but use the schema across agents rather than inside one. We also keep the schema deliberately small, because OaK's repeated judge-and-repair calls are a known cost.

On the cross-modal side, two papers show that most visual content is irrelevant and can be dropped. PSG prunes visual scene-graph nodes against a parsed sentence graph, gaining 1.84 BLEU over VALHALLA on Multi30K. HAP keeps 5.6% of visual tokens while preserving 99.1% of performance by weighting the few attention heads aligned with the prompt. Both prune by statistical alignment with text. We prune the image agent's reported entities by type reachability instead. That signal is available even when the question never mentions the visual entity, which is exactly the bridge-entity case.

InEx builds a cross-modal multi-agent critique loop around generative image editing. We avoid generative verification entirely, since edited images can introduce new errors.

Constraints-of-Thought is the closest precedent for pruning a language model's search with executable constraints before expansion. It also warns that pruning is only as good as constraint extraction, and this warning shapes our design. For us, the analogue of extraction error is mistyping a visual entity.

Research Program

We would build a team of four agents: a planner, a text agent over passages, a table agent, and a vision-language image agent. Each tool-using agent is declared with the relations it can resolve. For each benchmark domain, a class graph of roughly 30 to 80 classes is drafted once by a language model from table headers and passage categories. It is checked for consistency with a reasoner in the manner of OaK and then frozen, so construction cost is paid per domain, not per query. The planner types the question's answer class $c^\*$.

Agents emit only typed triples. The image agent's output passes through a filter that keeps only detected entities whose class lies on some path to $c^\*$. This is our version of density-gap pruning.

To guard against over-pruning, a link is cut only when every entity class in $S_t$ was assigned with confidence above a threshold $\tau$; otherwise the link stays open. Sweeping $\tau$ traces a curve of cost against accuracy, and we measure directly how often the gold evidence source was wrongly pruned. We would also analyse where the method fails. We would stratify errors into mistyped entities, missing relations in the schema, and reasoning errors downstream of correct pruning. Only the first two are attributable to the ontology.

Baselines

The naive reference point is a fully connected team of the same four agents, where every agent receives every message for a fixed number of rounds. It shows how much communication is wasted. Beside it we run a single vision-language model with ReAct-style prompting over the same tools. TravelPlanner reports only a 0.6% final pass rate for GPT-4-Turbo with ReAct in its two-stage setting, but that is a different task, so we measure this baseline ourselves.

The strongest existing method on this exact question is M3Prune. It targets the same problem, pruning multimodal multi-agent communication, and reports gains on MultimodalQA. The file gives only its macro figures, a 9.4% accuracy improvement and a 15.7% token reduction, and no reported result was found on Lacuna for M3Prune's per-benchmark MultimodalQA F1. We will therefore reimplement it on our identical agents and backbone, give it the same training questions and the same hyperparameter search budget, and report both its F1 and its tokens.

For absolute reference on MultimodalQA cross-modal questions, MultiModalQA reports 51.7 F1 for ImplicitDecomp, 38.2 F1 for AutoRouting, and 90.1 F1 for humans. Beating M3Prune means matching or exceeding its cross-modal F1 at fewer tokens per question, or exceeding its F1 at equal tokens, with confidence intervals from question-level bootstrap resampling.

Three ablations isolate the key ingredients. The first is post-filtering: all agents run, and messages that violate types are discarded afterward. It separates proactive pruning from mere type checking. The second is one-shot ontology pruning, which applies reachability only at round zero. It separates dynamic reachability from static routing. The third is random pruning at a matched link budget, together with a shuffled-type ontology, which tests whether the ontology's content matters rather than sparsity alone.

Why not just…?

Why not just train a router that picks which agents to call? This is the most serious threat. OG-MAR already uses a DeBERTa topic classifier to decide what to retrieve, and a well-trained router might capture most of the savings. Our bet is that a router deciding before any evidence arrives cannot anticipate bridge entities. The test is to stratify MultimodalQA questions by whether the modality of the second hop can be predicted from the question text alone. If a trained router matches us on the unpredictable stratum as well, the dynamic-reachability claim fails, and we would report the method as a cheaper, training-free router rather than a better one.

Why not just use M3Prune's learned pruning? Learned pruning needs training data per team and task, and it prunes by correlation. It cannot guarantee that a link carrying the only path to the answer type is preserved, nor that a type-impossible link is cut in round one. The head-to-head comparison above, plus a transfer test in which M3Prune is trained on one domain split and run on another while our schema is rebuilt from headers, would show whether symbolic pruning's lack of training is an advantage in practice.

Why not just run everything and post-filter? Post-filtering discards invalid statements but still pays for generating them. Worse, the invalid messages may already have entered other agents' context, as M3Prune's finding that fully connected networks lose accuracy to noise suggests. Evidence from constrained rule mining (TaSRM) shows proactive pruning dominating post-filtering in cost. The post-filter ablation tests whether that transfers when each "candidate" is an expensive model call.

Why not just use one vision-language model with visual token pruning? HAP shows that a single model can drop most visual tokens cheaply. But its pruning depends on the prompt mentioning what to look for, and a single model does not decompose table-then-image hops. The single-model baseline, given HAP-style token pruning at matched visual budget, measures whether decomposition is needed at all on these questions.

Proposed Experiments

The primary benchmark is MultimodalQA, reporting cross-modal and overall F1 and exact match, tokens per question, agent calls per question, and wall-clock time. As a secondary benchmark, ScienceQA serves as a regime where answer types are coarse (multiple choice) and reachability should give little leverage. Relevant reference points there are 75.17% for GPT-3 with chain-of-thought and 88.40% for humans, as reported in ScienceQA. Near-zero gain there would support the claim that the benefit comes from typed multi-hop structure, not from extra prompting. To check whether typed, reachability-filtered image reports reduce object hallucination in the image agent's statements, we would run its component on POPE, where InEx reports an 8.9% accuracy gain for LLaVA-1.5 on the MSCOCO subset.

Four checks decide the outcome. The first is the bridge-entity stratification against the trained router. The second is the over-pruning rate, the fraction of questions whose gold evidence source was cut, plotted against $\tau$. The method is only credible if this rate stays below the accuracy margin it buys. The third is the matched-token comparison with reimplemented M3Prune. The fourth is the shuffled-type control. If the shuffled ontology performs as well as the real one, the gains come from sparsity, not semantics, and the core claim is false.