When a vision-language model is asked a hard question about a diagram or chart, sampling it several times produces several reasoning chains that disagree. Each chain contains small factual claims: "the axis tops out at 60," "the triangle is isosceles," "so the area is 24." Today these chains are combined by majority vote on the final answer, which throws away the intermediate claims. That is exactly where the documented failures live. Models misread elementary chart details, and they read the figure correctly but then deduce wrongly. This note proposes to keep the claims and turn them into a signed graph, a network whose edges are marked positive (two claims support each other) or negative (they contradict each other). We then search that graph for the largest internally consistent group of claims, using algorithms from the community-detection literature built for exactly this "who agrees with whom" structure. The final answer is regenerated from that consistent group alone.
The two-sentence version is this. Sampled reasoning chains are mined for atomic claims, and contradictions among those claims are written as negative edges. A tolerant structural-balance search then picks out the coherent set of facts that the model "believes" across samples, so a chain with correct perception but bad arithmetic can be salvaged, and two chains that reach the same wrong answer through different misreadings do not reinforce each other. A secondary, cheaper claim is that the spectral "frustration" of this claim graph tells us, before answering, which questions the model is likely to get wrong.
Core Claim
The motivating evidence is that multimodal failures sit in the intermediate steps, not only in the final answer. On MATH-Vision, a GPT-4V error analysis attributes 42.2% of failures to reasoning errors with correct recognition and 31.9% to vision recognition errors (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). The same paper reports that chain-of-thought prompting does not reliably help multimodal models. On Chart101, models get elementary readings wrong, and fewer than 25% of charts have all of their elementary questions answered correctly, even for the best models (Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details). Both findings suggest that errors are local and partly independent across samples. If so, they should show up as contradictions between claims. Answer-level voting cannot see those contradictions; a claim-level consistency search can.
The method side comes from signed-network community detection. Under structural balance theory, a signed graph is balanced if its nodes split into two factions with only positive edges inside each faction and only negative edges between them. We use the split "accepted claims" versus "rejected claims." Finding the largest balanced subgraph is NP-hard. Finding large balanced subgraphs in signed networks (Timbal) attacks it by pruning nodes that raise the smallest eigenvalue $\lambda_1$ of the signed Laplacian. That eigenvalue is zero exactly when the graph is balanced, so it acts as a frustration meter. Scalable Algorithm for Finding Balanced Subgraphs with Tolerance in Signed Networks relaxes strict balance with a tolerance parameter $\beta$, because strict balance on noisy real data yields uselessly small subgraphs. It searches with local insert, delete, and side-flip moves. We reuse both techniques.
What is new is the object they run on. These papers balance social networks with millions of edges. We balance per-question graphs of a few hundred model-generated claims, and we use the balanced faction to condition generation. Because the graphs are small, NP-hardness is not a practical barrier: we can solve exactly with an integer program and use the heuristics to test whether they are good enough. This is a contribution about representation, not about scaling.
The objective follows the tolerant-balance idea. For a candidate claim set $S$ and a side assignment $\sigma: S \to \{\text{accept}, \text{reject}\}$, we maximize
$$ \sum_{v \in S,\ \sigma(v)=\text{accept}} s_v \;-\; \beta \sum_{(u,v) \in E_S} w_{uv}\, \mathbb{1}[\text{edge } (u,v) \text{ violates balance under } \sigma], $$
where $s_v$ is the support of claim $v$ (how many sampled chains assert it), $E_S$ is the set of edges among claims in $S$, $w_{uv}$ is the checker's confidence in the edge, and $\beta$ trades coverage against tolerated inconsistency. A positive edge violates balance if it crosses sides; a negative edge violates balance if both of its endpoints are accepted.
We do not count negative edges among rejected claims. Two wrong readings can contradict each other, so the "reject" side is not a faction in the social sense. This asymmetry is the one deliberate departure from classical balance, and an ablation tests it against the symmetric version.
Research Program
For each question we sample $K$ chains (default 16) from the multimodal model. A text-only language model splits each chain into atomic claims, and claims are canonicalized so that quantities and relations merge across chains. This follows the claim-extraction step that VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering uses for dataset filtering. We apply it at inference time instead.
Edges come from three sources, and their reliability is kept separate. Deterministic checks produce negative edges: two different values for the same quantity, an ordering violation such as a minimum exceeding a maximum, or an arithmetic step that fails when executed. The execution check follows the symbolic decomposition idea of Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA. A text-only checker produces entailment edges (positive) and contradiction edges (negative), each with a confidence score. Optionally, a vision tool such as OCR or a detector provides anchor claims, which become query nodes. With anchors, the problem becomes community search: find the best balanced faction that contains the anchors, the localized formulation that the community-search literature studies.
We solve the objective three ways: exactly with an integer program, with the tolerant local search, and with Timbal-style spectral pruning. The model then answers a final time, conditioned on the image and the accepted claims only.
Separately, we compute $\lambda_1$ of each question's signed Laplacian, normalized by graph size. We test whether it predicts error well enough to support abstention. This is a measurement study and needs no new training.
Baselines
The naive reference point is greedy single-sample decoding of the same model with standard chain-of-thought. On MATH-Vision, the strongest reported closed-model figure in the file is GPT-4o at 30.39% accuracy, against 68.82% for humans (MATH-Vision). On GAOKAO-MM, GPT-4V reports 48.1% (GAOKAO-MM). We rerun these numbers ourselves for the models we use, since API versions drift.
The strongest competitor is self-consistency: majority vote over the final answers of the same $K$ chains. It is the real threat. It costs nothing beyond sampling, and any benefit of our method could simply come from having more samples. No reported result was found on Lacuna for self-consistency on MATH-Vision, Chart101, or GAOKAO-MM, so we run it ourselves. To make the comparison fair on compute, we also run self-consistency with enough extra samples to match our total token cost, including extraction, checking, and the final regeneration. Our method beats it only if it wins at matched tokens on MATH-Vision and on Chart101 collective accuracy.
A second baseline is visual prompting with retrieved exemplars, DCoT (DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models). It reports improving LLaVA-1.5-13B from 36.1% to 38.4% on MM-Vet and gains of 2.47% on MM-Bench. We rerun it on our benchmarks with the same base model.
A third baseline is multi-agent debate with tool checking (Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System). It is run with the same vision tool and a matched number of model calls. No reported result was found on Lacuna for it on reasoning benchmarks.
Four ablations isolate the key ingredient, which is the use of contradiction structure. The first keeps claims supported by more than half the chains, with no graph. The second drops negative edges and clusters only the positive ones. The third uses symmetric classical balance instead of our asymmetric objective. The fourth replaces the balanced faction with correlation clustering, which allows many groups, and takes the best-supported group. If the frequency filter matches the full method, the graph adds nothing, and we will report that.
Why not just…?
Why not just use self-consistency voting? Voting pools chains that agree on an answer even when they got there through incompatible readings. It also discards chains whose perception was right but whose final step failed. The MATH-Vision error split, with substantial shares of both recognition and reasoning errors, suggests that both cases are common, but it does not prove that they are independent across samples. If sampled errors are highly correlated, voting and our method will converge. The matched-token comparison settles this. We will additionally stratify results by the per-question disagreement rate, since that is where any advantage should appear.
Why not just ask a verifier model to judge each chain? A chain-level judge sees one chain at a time and has to detect errors that are often only visible as conflict with other samples. The MATH-Vision finding that intermediate textual steps compound errors also argues against trusting a single long rationale. We include an LLM-judge reranker at matched calls. If it ties our method, the cross-sample structure is not doing work.
Why not just convert the image to text and reason over the text? GeoSQA tested this with human-written diagram descriptions, and models stayed near chance: 26.22% for the best model against a 25% random baseline (GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level). However, those were pre-LLM models, so the evidence is weak. Our ablation that removes the image at final regeneration directly measures how much the accepted claims alone carry.
Why not just run unstructured debate? Debate costs many calls and is sensitive to how the supervisor is prompted (Yu et al.). Our resolution step is a deterministic optimization once the edges exist. The matched-call comparison tests whether that matters for accuracy and not just for cost.
Proposed Experiments
The main experiment covers MATH-Vision (all 3,040 problems, reported per subject and difficulty level), Chart101 (elementary accuracy and the collective all-correct-per-chart rate), and the mathematics and physics subsets of GAOKAO-MM. We use one closed and one open model, with $K$ set to 4, 8, and 16. Chart101 is the cleanest test, because many of its contradictions are deterministic numeric conflicts. It also includes the known title and grid-line perturbations from Plot Twist, so we can ask whether balance search recovers accuracy lost to distractors.
Three checks are decisive. The first is whether the method beats token-matched self-consistency by a margin larger than seed variance on MATH-Vision and Chart101 collective accuracy. The second is whether it beats the claim-frequency and positive-only ablations; if not, the signed structure is unnecessary. The third is whether the normalized $\lambda_1$ predicts error, measured by the area under the ROC curve for error detection, better than answer-vote entropy. If the first two checks fail but the third succeeds, the honest conclusion is that signed claim graphs are a diagnostic for unreliable answers, not a repair method.
We will also report agreement between exact and heuristic solutions on these small graphs. This tells us whether the scaling machinery of Timbal and tolerant balance search is needed here at all.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Finding large balanced subgraphs in signed networks
- Scalable Algorithm for Finding Balanced Subgraphs with Tolerance in Signed Networks
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level