Fusing Sampled Evidence Boxes Before a Vision-Language Model Reasons

Vision-language models often answer chart, document and diagram questions wrongly because they read the wrong part of the image. A popular fix is to make the model look first. The model outputs a bounding box (a rectangle around the region it thinks matters), a script draws the box on the image, and the model then answers with the marked image in front of it. This helps, but it trusts a single guessed rectangle. If that one guess is off, every later reasoning step inherits the error.

Object detection hit the same problem years ago and solved it. Instead of keeping only the highest-scoring box and deleting its overlapping rivals, weighted boxes fusion averages the coordinates of all overlapping candidates, weighting each by its confidence. We propose to treat a vision-language model's own sampled "where to look" boxes as an ensemble of weak detectors. We fuse them with calibrated weights, keep separate clusters when a question needs several regions, and use the spread of the samples to decide whether to draw any box at all. The bet is that spending compute on cheap box samples, a few dozen tokens each, buys more accuracy than spending it on extra full reasoning chains.

Core Claim

Sampling $N$ box proposals from the model, clustering them, and fusing each cluster into one evidence region yields better grounding than the single greedy box used by look-first prompting. The improved grounding carries through to answer accuracy, but only on questions where the failure is perceptual. At matched generated-token budgets, fusion in box space should beat majority voting in answer space on perception-heavy benchmarks. It should not help on reasoning-dominated ones.

This claim has three parts that existing work does not combine:

  1. Confidence-weighted fusion of the model's own box samples. Prior box-guided reasoning methods pick one box. Prior fusion methods never touch a language model.
  2. Calibrated weights. The weights are calibrated against how well a box covers the true evidence, not taken from raw token probabilities.
  3. Dispersion as a gate. How widely the sampled boxes disagree is treated as a signal of perceptual uncertainty and acted on.

Where This Sits

The motivating evidence comes from three places:

  • Benchmarks show a large perception problem. MATH-Vision attributes 31.9% of GPT-4V failures to vision recognition errors, though reasoning errors (42.2%) are larger still. Plot Twist shows models misreading elementary chart values and leaning on titles, with accuracy collapsing to the 30% range when titles mislead. Both tell us looking in the right place is a real bottleneck, while warning that it is not the only one.
  • Box-guided prompting works. DCoT has the model predict one box, draws it in red, and re-prompts. Doc-CoB has the model pick layout-parser boxes by ID and blurs everything else. CoTBox-TTT crops around one predicted box and adapts soft prompts at test time for medical questions.
  • All three keep a single hypothesis or make a keep-or-blur choice. That is the winner-takes-all habit detection abandoned. We keep their re-prompting interface and replace the single box with a fused one.

The method comes from detection ensembling:

  • MoCaE shows that coordinate averaging is distorted when some experts are overconfident. It fixes this by isotonic-regression calibration of scores against IoU (intersection-over-union, the overlap between a predicted and a true box). We reuse this calibration step almost verbatim. The difference is the target: overlap with an evidence region rather than with an object.
  • Union-over-Intersections shows that taking the geometric union of partial views can beat averaging when each proposal sees only part of an object. That matters here because missing evidence is worse than including extra context, so we test a union-style fusion alongside averaging.

Neither paper interfaces with language generation, and that is the gap we fill.

Two hallucination papers constrain the design:

Both fuse in language space. Our argument is that geometry is the cheaper and more direct place to reach consensus about where evidence lies.

Research Program

The pipeline has four stages.

Sampling. Given an image and question, we prompt the model $N$ times, varying temperature and the wording of the look-first instruction, to emit one or more boxes. With $N=8$, the box-only outputs cost roughly as many tokens as a single short reasoning chain.

Clustering. Boxes are clustered by pairwise IoU as in weighted boxes fusion. Questions about one value produce one dominant cluster. Comparative questions ("which bar is taller") should produce several clusters, and we keep every cluster supported by at least a minimum fraction of samples, rather than merging everything into one. This mirrors detection's dense same-class problem, where merging two adjacent objects is the classic error.

Fusion and calibration. Each cluster $k$ is fused into one box:

$$\hat b_k = \frac{\sum_{i \in k} w_i\, b_i}{\sum_{i \in k} w_i}, \qquad w_i = g(s_i),$$

where $b_i$ is a sampled box's four coordinates and $s_i$ is a raw score. The raw score is either the mean log-probability of the coordinate tokens or the fraction of samples landing in the cluster. The function $g$ is an isotonic regression fit on a held-out split, mapping the raw score to expected evidence coverage. A union variant replaces the weighted mean with the union of the cluster's highest-weight members.

Gating. We measure dispersion as one minus the mean pairwise IoU within the top cluster.

  • When dispersion exceeds a threshold tuned on validation data, the system draws no box and answers from the raw image, because a confidently wrong mark is worse than none.
  • Otherwise the fused boxes are drawn, DCoT-style, and the model produces one reasoning chain.

Analysis. The analysis asks whether accuracy gains are carried by better grounding. We record evidence-region IoU per question, and test whether accuracy changes are statistically explained by IoU changes rather than by the mere presence of a red rectangle.

Baselines

Naive reference. The plain model answers from the unmarked image with a single chain-of-thought. We use LLaVA-1.5-13B, the model DCoT reports on, and InternVL2-8B, the model Doc-CoB reports on. Everything else is measured as a difference from this.

Single-box look-first prompting. This is the strongest direct competitor: DCoT reimplemented with its greedy box and red-box redraw. Our method changes only how the box is produced. DCoT reports LLaVA-1.5-13B reaching 38.4% on MM-Vet against 36.1% for the base model, and an average +2.47% on MM-Bench. We will reproduce these numbers before comparing. Both methods use the same prompts for the final answering turn and the same in-context examples. Beating DCoT means a gain on MM-Vet and the chart and text benchmarks that exceeds seed variance at an equal answering budget.

Doc-CoB on documents. On document benchmarks the strongest system is Doc-CoB. It reports InternVL2-8B rising from 36.33% to 80.05% on DeepForm. We run it with the same layout parser and model. Our variant samples ID selections $N$ times and fuses the selected boxes, including continuous refinement beyond parser boundaries. Matching Doc-CoB while removing its dependence on parser boundaries would already be informative. Exceeding it on questions where the parser mis-segments the evidence is the target.

Answer-space self-consistency. We sample several full reasoning chains on the unmarked image, and also on a single-box marked image, then majority-vote the answers. We found no reported result on Lacuna for self-consistency on these benchmarks, so we run it ourselves. It is matched to our method in total generated tokens and number of forward passes, at two budget levels.

Ablations isolating the key ingredient.

  • A random single sample from the $N$ boxes separates sampling from fusion.
  • Uncalibrated fusion with raw weights tests the MoCaE-style calibration.
  • Average versus union fusion tests the recall argument.
  • Fusion without the gate tests the dispersion signal.
  • An oracle best-of-$N$ box upper-bounds what selection alone could achieve.
  • A deliberately shifted box measures how much accuracy is lost to bad marks.

Why not just…?

Why not just use self-consistency over answers? This is the real threat. Voting over answers already averages out bad looks implicitly. Our counterargument is cost and placement. A box sample costs a few dozen tokens while a chain costs hundreds, and MATH-Vision reports that chain-of-thought sometimes lowers accuracy for these models, so more chains can mean more drift. The matched-budget comparison settles it. If self-consistency equals box fusion at equal tokens on the perception-heavy subsets, the proposal's central claim is false, and we would report the two as complementary at best.

Why not just get the box from a real detector or grounding model? Open-set detectors such as GroundingDINO, used as a tool in the multi-agent system, localize named objects well. However, "the evidence for this question" in a chart is often an axis tick, a legend entry or the gap between two bars, which is not a nameable object. We include a detector-box baseline on TextVQA and the chart sets, and also add detector boxes as extra ensemble members. If detector boxes alone win, fusion still matters only as a way to combine them with the model's own guesses.

Why not just crop or blur generously around a single box? A padded single box is a crude version of the union variant, so we test it with padding tuned on validation data. Padding cannot fix a box centered on the wrong bar, and it cannot represent two separate evidence regions. The gap between padded-single and multi-cluster fusion on comparative questions measures exactly this.

Proposed Experiments

Grounding quality, where ground truth exists. On TextVQA and ST-VQA, the answer's OCR token boxes give the evidence region, as exploited by LaAP-Net. On DocVQA the answer string can be located by OCR match. On Chart101 from Plot Twist, synthetic plots allow rendering the true mark positions. On these sets we report evidence IoU and coverage for greedy, fused, union and oracle boxes.

Downstream accuracy. We measure accuracy on the same sets plus ChartQA, MM-Vet, DeepForm, and MathVista. For scale, GPT-4V scores 49.9% on MathVista against 60.3% for humans, with open models between 20% and 26%.

Decisive checks. Four results would settle the proposal:

  1. Grounding. Fused boxes must cover the evidence better than greedy boxes.
  2. Mediation. Per-question accuracy gains must track IoU gains.
  3. Budget. Box fusion must beat answer self-consistency at matched tokens on Chart101, TextVQA and ChartQA.
  4. Dispersion. Dispersion should predict wrong answers, measured by AUROC on held-out questions, and the gate should recover accuracy that ungated fusion loses on high-dispersion questions.

Negative control. On MATH-Vision we predict small or no gains in categories like transformation geometry, where GPT-4o is near 18% and failures are deductive, and larger gains on figure-reading categories. Finding the opposite pattern would mean the method helps for reasons other than grounding, and we would say so.