Multi-agent pipelines can sharply improve how vision-language models solve diagram-heavy math and physics problems. These pipelines split the work among separate model calls: one reads the figure, one writes the constraints down as text, and one plans and computes the solution. The best recent example, PRISM, lifts a small Gemini model from 37.0% to 64.5% on SeePhys-mini. It also runs every question through the full pipeline, which wastes tokens and can hurt accuracy on easy, text-heavy questions. The natural fix is to call the expensive specialists only when they are needed. Existing ways of deciding "when needed" have a blind spot. They watch for disagreement between agents or for low self-confidence, and the most damaging multimodal failure is a confident, unanimous one. The model reads the chart title instead of the chart, every sampled answer agrees, and nothing triggers.
This note proposes a different signal for when to escalate. Before calling any specialist, ask the cheap main model the same question twice more: once with the image removed and once with a semantically irrelevant change to it, such as added grid lines or a swapped title. Suppose the answer is unchanged when the image disappears, on a question whose solution depends on the figure. Then the image is not actually participating in the answer, and that is exactly when visual specialists should be recruited. The architecture borrows the shape of online communities, where a small active core does most of the work and a long periphery contributes only occasionally. One core reasoning agent handles most questions alone. Narrow perceptual and computational specialists are recruited individually, as the probes indicate.
Core Claim
A trigger based on counterfactual image probes catches confidently wrong, shortcut-driven answers that disagreement-based and self-consistency-based triggers miss by construction. Combined with recruiting specialists one at a time rather than switching on the whole pipeline, it should recover nearly all of full-PRISM's accuracy at a fraction of its token cost. The claim is falsifiable. If a self-verification cascade in the style of Single-agent or Multi-agent Systems? Why Not Both? catches the same errors at the same cost, the counterfactual probes add nothing and the idea fails.
What the Prior Work Establishes
Three findings motivate the proposal by showing that shared, confident visual errors are common.
- Plot Twist probes elementary chart reading. Accuracy reaches 79% when chart titles carry semantic clues and falls to the 30% range when titles are removed or made misleading. Adding grid lines drops PaLI-3 from 29.5% to 21.3%. This is direct evidence that answers often ride on text priors rather than on the pixels.
- MATH-Vision attributes 31.9% of GPT-4V's errors to visual recognition and 42.2% to reasoning. It also finds that chain-of-thought (step-by-step written reasoning) is erratic in this multimodal setting.
- PRISM reports that the gap between Pass@k (whether any of k samples is correct) and Majority@k (whether the majority vote is correct) widens as k grows. Sampling the same model more simply repeats the same perceptual mistake.
Two lines of work constrain the design.
- Single-agent or Multi-agent Systems? Why Not Both? measures multi-agent systems using 4x to 220x more prefill tokens than a single agent. It also finds they underperform a single agent on easy tasks, because extra critique triggers false corrections. Its fix is a binary routing or cascade between one agent and the full system, tested only on text, so any proposal here must beat that switch, not merely match it.
- Peacemaker or Troublemaker shows that debating agents abandon correct answers to agree with a confident peer. A central judge cuts the disagreement-collapse rate from about 81% to 41% on CommonsenseQA. That result argues for a single core decision-maker rather than an egalitarian debate.
The two closest systems supply methods and define the gap.
- DART runs several different vision-language models and recruits grounding and text-reading tools only when their answers conflict. It reports 65.5% on MMMU. It needs every base model run on every question, and it cannot fire when all models share an illusion.
- PRISM supplies the specialist agents we reuse: a grounding agent that runs OpenCV code to locate figure elements, a step that writes an enriched text caption, and a planner and solver that call SciPy. PRISM itself names "adaptive routing" as open work.
The participation-inequality literature plays a narrower role: it is a lens, not a theory we test. Cunha et al. find that concentrated activity, a high Gini coefficient (a 0-to-1 measure of how unevenly activity is spread), predicts community survival rather than threatening it. Segal et al. show that predicting the moment a Galaxy Zoo volunteer is about to disengage, and intervening only then, yields 19.6% more contributions than random intervention. We borrow two ideas from this work: measure who actually contributes, and intervene at a predicted failure point rather than constantly.
Research Program
Stage 1: measure contribution. We first measure how unequal agent contribution is inside a PRISM-style pipeline. For each question in SeePhys-mini and MATH-Vision, we run leave-one-specialist-out ablations. For every specialist, we record whether removing it flips the final answer from right to wrong. From this we compute the per-question and pooled Gini coefficient of causal contribution across specialists. We have two hypotheses:
- Contribution is heavy-tailed. On most questions the core alone suffices or a single specialist matters.
- Which specialist matters varies with the type of question.
If contribution turns out to be flat, with every specialist mattering on every question, then per-specialist recruitment is pointless. The design would collapse to Gao et al.'s binary switch, and we would report that.
Stage 2: build the escalation predictor. The core agent answers each question three ways:
- with the original image;
- with the image replaced by a blank;
- with an irrelevant perturbation applied, such as grid lines, a different marker style, or a neutralized title.
We also take a small self-consistency sample and record the agreement rate. A small classifier, trained on a held-out split, reads several features:
- whether the answer survives image removal;
- whether it survives perturbation;
- the self-consistency rate;
- simple features of the question text.
The classifier predicts two things. The first is whether the core's answer is wrong. The second is which specialist's ablation, as measured in Stage 1, would fix it. Image-dependence alone is not a valid signal on vision-optional questions, where answering from text is legitimate. The classifier therefore has to learn the interaction between "the answer ignores the image" and "this question needs the image." It is not given a fixed rule.
Stage 3: deploy the core-periphery system. The core answers alone unless the predictor fires, in which case only the predicted specialists are recruited. Recruiting the grounding agent alone is one example; recruiting the solver-side critic for suspected reasoning errors is another. The core then re-answers with the specialists' evidence. We expect image probes to address recognition and shortcut errors but not MATH-Vision's 42.2% pure reasoning errors. For those, the self-consistency feature and a solver-side verification check carry the signal. We will report the two error classes separately rather than claim one trigger covers both.
Baselines
The naive reference is the core model answering directly with one call, alongside the same model with chain-of-thought. These give the cost floor. MATH-Vision's finding on chain-of-thought instability makes it plausible that the chain-of-thought variant will be worse than direct answering.
The sampling baseline is Majority@k with the core model, with k chosen so that its token cost matches our system's average. This tests whether spending the same budget on more samples is as good as spending it on targeted specialists.
The accuracy ceiling is full PRISM run on every question. PRISM reports 37.0% to 64.5% on SeePhys-mini with Gemini 3.1 Flash Lite, and 42.8% to 59.5% on MATH-Vision. We rerun it with our core model, specialists and prompts so that every system shares the same components. Beating it means matching its accuracy within one point at substantially lower average tokens.
The strongest cost-aware competitor is Gao et al.'s Agent Routing and Agent Cascade, applied with PRISM as the multi-agent arm.
- Agent Routing uses a difficulty classifier.
- Agent Cascade escalates only when an answer-verification check fails.
On text tasks they report up to 88.1% cost savings and accuracy gains of 1.1% to 12%. We found no reported result on Lacuna for either method on any multimodal benchmark. Both receive the same training split for their classifier or verifier as our predictor, and the same tuning budget for thresholds.
The disagreement-trigger competitor is a DART-style setup. It uses the same pool of open models DART used (QwenVL, MiniCPM, Ovis2) and recruits PRISM specialists only on conflict. DART's reported MMMU accuracy is 65.5%.
Two ablations isolate our ingredients.
- Counterfactual-feature ablation: we remove the blank-image and perturbation features, which reduces our predictor to a confidence-only trigger. This isolates the contribution of the counterfactual probes.
- Recruitment-granularity ablation: when the predictor fires, we always recruit the whole pipeline. This isolates the value of recruiting specialists individually.
An oracle router that knows which specialists would fix each question bounds what any trigger could achieve.
Why not just…?
Why not just use Gao et al.'s Agent Cascade on PRISM? This is the real threat. A good verifier might already reject shortcut answers. The cascade's verification step, however, inspects the answer and its reasoning. It never asks whether the image contributed. A confident answer read off a chart title passes a plausibility check. The decisive comparison is the recall of each trigger on core errors that are also highly self-consistent, measured separately on SeePhys-mini's Vision-Essential subset. If the cascade's recall there is within noise of ours, we concede the point.
Why not just sample more and vote? PRISM's own finding that the gap between Pass@k and Majority@k widens with k says that perceptual errors are not independent across samples. We test this directly with the cost-matched Majority@k baseline.
Why not just use disagreement across different models, as DART does? Model diversity helps only if the models do not share a blind spot. Plot Twist's title effect, along with collective accuracy below 25% on Chart101 across leading models, suggests they often do. DART also pays for all base models on every question. Our Chart101 title-swap experiment measures directly how often unanimous ensembles are wrong.
Why not just always run the full pipeline? PRISM itself reports regressions on simple text-heavy questions, and Gao et al. quantify the token premium. Full PRISM is our accuracy ceiling, not our target.
Proposed Experiments
The main benchmarks are SeePhys-mini, using its Vision-Essential and Vision-Optional split, and MATH-Vision, reported by difficulty level. We add Chart101 from Plot Twist as a stress test, using its original, removed and misleading titles and its grid-line perturbations. MMMU serves as a broad check for comparability with DART. Where API access allows, the core model is Gemini 3.1 Flash Lite, as in PRISM; an open vision-language model from DART's pool serves as a second core.
We report the following metrics:
- accuracy;
- mean prompt and output tokens per question;
- the area under the accuracy-versus-tokens curve as thresholds vary;
- escalation precision and recall against the oracle router;
- the Gini coefficient of specialist contribution from Stage 1.
Four results would decide the question.
- On Vision-Essential SeePhys questions where the core is wrong yet self-consistent, the counterfactual trigger should catch clearly more errors than both the Agent Cascade and the DART-style trigger.
- Overall, the system should reach full-PRISM accuracy within one point while recruiting specialists on a minority of questions.
- On Chart101, the escalation rate should rise when titles are made misleading, and accuracy after escalation should recover toward the with-title level.
- The recruitment-granularity ablation should cost tokens without gaining accuracy. That would confirm that unequal, per-specialist participation, not mere escalation, is doing the work.
If the cascade matches us on the first test, the counterfactual probe is unnecessary. If contribution in Stage 1 is flat, the core-periphery structure is unjustified. Either outcome is a clean negative result for the field.
References
- PRISM: Structured Decomposition for Multimodal Physics and Mathematical Reasoning
- Single-agent or Multi-agent Systems? Why Not Both?
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning
- Are All Successful Communities Alike? Characterizing and Predicting the Success of Online Communities
- Intervention Strategies for Increasing Engagement in Crowdsourcing: Platform, Predictions, and Experiments