Does the Model Read the Arrows? Topological Minimal Pairs for Diagram Reasoning

Vision-language models, which answer questions about images, now reach respectable accuracy on flowchart question answering. That accuracy may say little about whether they follow the diagram's structure. Flowchart labels carry strong everyday meaning: "preheat oven" usually comes before "bake". A model can therefore answer "can step A lead to step E?" from what the labels suggest, without tracing a single arrow. Our test is simple. Take a diagram, reverse or delete one edge so that the correct answer flips, and leave the labels, the layout, and almost every pixel unchanged. A model that reads the arrows flips its answer. A model that reads the labels does not. We pair these "topology-changing" edits with "topology-preserving" re-renderings, where the same graph is drawn with a different layout or renderer and the answer must stay the same. Together they separate sensitivity to structure from sensitivity to pixels. This gives a verifiable measure of whether a correct answer was actually derived from the graph.

The same pairs also serve as a training signal. Every pair comes from a known graph, so both the answer and every edge the model cites in its explanation can be checked exactly, with no fuzzy verifier. We propose to train an open vision-language model with a reward that pays only for getting both members of a pair right, together with an edge-level check on the stated path. We will then test whether this closes the gap between answer accuracy and structural faithfulness better than three alternatives: outcome-only reinforcement learning, more synthetic training data, and parse-then-search pipelines.

Core Claim

The claim has two parts. The first is that outcome accuracy on flowchart and graph benchmarks overstates topological reasoning. Specifically, a large share of correct answers on real-content flowcharts will not survive a single answer-flipping edge edit, and that share will be largest exactly where label semantics agree with the original edge direction. The second is that a pair-consistency reward, computed on a known graph, teaches models to condition on edges rather than labels. We expect this reward to raise pair-consistent accuracy and edge-valid explanations without the accuracy loss seen when faithfulness is enforced through softer preference training.

Four lines of work motivate this. FlowVQA builds flowchart question answering from WikiHow-style Mermaid scripts and finds a 15% GPT-4V drop when charts are flipped bottom-up, plus weak topological-task performance. Its scoring, however, is purely final-answer. Visual Graph Arena shows that models perform near random guessing on visual graph isomorphism and are sensitive to layout changes (ConvNeXt shortest-path accuracy fell from 82.4% to 73.3% when the layout changed). It proposes no fix and does not inspect reasoning. Plot Twist shows chart models leaning on titles: accuracy reaches up to 79% when the title contains the answer and falls to the 30% range when the title is removed or misleading. This is the chart analogue of the label shortcut we expect in flowcharts. Beyond Answer Correctness finds that 87.1% of BakLLaVA's correct ChartQA answers rest on hallucinated rationales. That work restricts evidence to flat tables and reports that nearly 47% of claims could not be verified at all. In our setting, evidence is a known edge set, so every claim is decidable.

Follow the Flow is the closest prior work. It asks models to output the supporting node-and-edge path on FlowExplainBench and solves this with a segmentation, Mermaid-extraction, and graph-search pipeline. Our work differs in three ways. First, it asks whether a stated path matches a human-annotated path; we ask whether the answer causally depends on the edges, which attribution alone cannot establish. Second, we train an end-to-end model rather than an external pipeline. Third, we include topology-preserving re-renders, so layout invariance is measured in the same instrument.

Research Program

The first component is the minimal-pair construction. FlowVQA diagrams are rendered from Mermaid scripts, so we can edit the source directly. For each test question about reachability, path existence, branch outcome, or node count, we search for the smallest edge edit that flips the ground-truth answer: an edge reversal, a deletion, or a rewiring that creates or breaks a cycle. We then re-render with the original layout engine. Edits are also classified by whether they contradict the label semantics. An annotator or a text-only model is asked, given only the labels, which order is plausible. This classification lets us measure the label shortcut directly.

For topology-preserving pairs, the same script is rendered through different engines (Mermaid, Graphviz, PlantUML), in different orientations, and with different layout algorithms. We follow the multi-renderer approach that FlowGen showed transfers to real and hand-drawn flowcharts. FlowGen's controllable generator, which varies graph order, branching, and nesting, supplies synthetic training pairs. On these, labels can be meaningful, shuffled, or random strings.

The second component is the metric. For a question set with pairs $(x, x')$ whose gold answers are $y, y'$, the pair-consistent accuracy is the fraction of pairs where the model answers both correctly. We define the topological faithfulness gap as

$$\Delta_{topo} = \frac{\#\{\text{correct on } x \text{ but wrong on its edited partner } x', \text{ or citing an edge not in } E\}}{\#\{\text{correct on } x\}},$$

where $E$ is the true edge set of $x$. This is deliberately parallel to the chart faithfulness gap of Beyond Answer Correctness, so the two numbers can be compared. It replaces their unverifiable table claims with exact edge checks, and it adds a causal component that their definition lacks. It is also a visual analogue of the counterfactual interventions in RFEval, which injects flawed text prefixes. We intervene on the input structure instead, which is closer to the multimodal perturbations of the medical VLM faithfulness study.

The third component is training. We fine-tune Qwen2.5-VL-7B, the model family used by FlowGen and by Perception-Judge, using group relative policy optimization (GRPO), a reinforcement learning method that scores several sampled answers against each other. The reward has two parts. The pair part is 1 only when the rollouts on both $x$ and $x'$ are correct, and 0 when only one is. This makes answering from the labels alone unprofitable by construction: a label-driven model gives the same answer to both members of a flipping pair, so it can never collect this reward. The trace part is the edge precision of the path the model states in a constrained format, checked against $E$. This is a soft, reward-level version of the hard validity guarantee that Decoding on Graphs enforces with trie-based logit masking on text knowledge graphs. We cannot mask at decoding time, because the graph is not available symbolically from the pixels at inference.

The motivation for this reward design comes from two findings. RFEval reports that outcome-rewarded reinforcement learning degrades faithfulness. rsbench shows outcome loss driving concept accuracy down to 10–20% while label accuracy stays around 80%. Both results constrain the design against outcome-only rewards. Perception-Judge supplies the method precedent: its GRPO with a ranked counterfactual reward raised perceptual error detection from 69.5% to 85.7%.

Baselines

The naive reference points are the untuned Qwen2.5-VL-7B and a text-only model given only the node labels and the question. The text-only model is essential. Its accuracy on original FlowVQA questions estimates how much of the benchmark is answerable from labels alone, and on flipping pairs it is guaranteed to fail one member, so it anchors the bottom of pair-consistent accuracy. We also run GPT-4V/GPT-4o-class models with few-shot chain-of-thought prompting. The reported reference is a peak FlowVQA accuracy of 68.42% for GPT-4V with few-shot CoT in FlowVQA. We expect that accuracy to shrink substantially once converted to pair-consistent accuracy.

The strongest data-centric competitor is FlowGen-style supervised fine-tuning: training the same 7B model on the same pool of synthetic flowcharts with standard answer and parse targets, but without pairing. This is the most serious competitor because FlowGen reports that it lifts Qwen2.5-VL-3B FlowVQA parsing F1 from 12.7% to 51.3%. FlowGen also finds that random-string labels still yield gains, which suggests plain synthetic data may already teach connectivity. It receives exactly the same diagrams we use, only unpaired, and the same number of gradient steps.

The strongest pipeline competitor is FlowPathAgent from Follow the Flow, which reports 77.20% attribution F1 on FlowExplainBench. We run it unmodified on our paired sets, since a pipeline's answer on the edited image depends only on how well it parses that image.

The strongest faithfulness-training competitor is Progressive Domain Alignment from Beyond Answer Correctness: staged fine-tuning plus faithfulness-aware preference optimization. We adapt it by replacing its table verifier with our exact edge check. It cut the chart faithfulness gap from 87.1% to 35.7% while accuracy moved from 77% to 75.5%. We also run outcome-only GRPO with the same rollouts, sampling, and step budget as our method.

Two ablations isolate the key ingredient: GRPO with the trace reward but no pairing, and GRPO with pairing but no trace reward. Every trained method receives the same base model, the same diagram pool, matched rollout and token budgets, and the same held-out hyperparameter search of equal size.

Beating the field means three things. On edited FlowVQA pairs, our model must achieve higher pair-consistent accuracy and lower $\Delta_{topo}$ than both FlowGen fine-tuning and adapted Progressive Domain Alignment, by a margin exceeding seed variance over three seeds. It must not lose unpaired FlowVQA accuracy relative to outcome GRPO. And it must match or exceed FlowPathAgent on the FlowGen Hard nested subset and on degraded scans.

Why not just…?

Why not just prompt a frontier reasoning model with chain-of-thought? Existing evidence argues against this. Visual Graph Arena found GPT-o1 and Claude 3.5 Sonnet near random on isomorphism, along with anomalies where easier path queries were answered worse than harder ones. RFEval found 49.7% of reasoning traces unfaithful. Neither result, however, measures label shortcuts in flowcharts specifically, so we test this directly. If frontier chain-of-thought already achieves low $\Delta_{topo}$ on semantics-contradicting edits, the diagnostic finding weakens and the training contribution becomes a matter of efficiency rather than capability.

Why not just parse the diagram into Mermaid and run breadth-first search? This is a real threat. A pipeline is faithful by construction whenever the parse is correct, and on clean, simple FlowVQA charts it may well win. Its weakness is the parse itself. FlowGen reports GPT-4o at only 12.0% F1 on nested subgraphs, and Follow the Flow acknowledges that errors from its early vision stages propagate. We therefore report results stratified by nesting depth and image degradation. We will state plainly that an end-to-end model is only justified in the strata where it beats the pipeline. Our metric, in any case, evaluates pipelines and end-to-end models on equal terms.

Why not just add more synthetic flowcharts, possibly with random labels? FlowGen's random-label result suggests that more data might produce topological reading on its own. The unpaired FlowGen fine-tuning baseline on identical diagrams settles this. If it matches our $\Delta_{topo}$ on real-content, semantics-contradicting edits, then pairing adds nothing, and we would report that finding as the result.

Why not just apply faithfulness preference optimization with a graph verifier? This is the adapted Progressive Domain Alignment baseline. Its preference pairs reward correct explanations, but they cannot penalize an answer that is right for label reasons while citing real edges. Only the causal pair penalizes that case. The pairing-only and trace-only ablations show which component carries any improvement.

Proposed Experiments

The diagnostic study comes first. We will evaluate all baselines on FlowVQA pairs with topology-changing edits and on re-rendered pairs, and report $\Delta_{topo}$ split by whether the edit agrees or conflicts with label semantics. The decisive diagnostic check is whether the gap is concentrated on semantics-conflicting edits. If it is, the gap is a label shortcut rather than general perceptual noise.

The training study follows. We will train our method, the baselines, and the ablations under matched budgets, then evaluate on four targets: held-out FlowVQA edited pairs; FlowExplainBench attribution F1 for comparison with the 77.20% FlowPathAgent figure; the FlowGen Hard subset and degraded scans; and FlowLearn. On FlowLearn, a FlowGen-fine-tuned Qwen2.5-VL-7B is reported at 60.1% F1 in FlowGen.

The transfer check uses Visual Graph Arena's layout-varied shortest-path and cycle tasks, with no training on them. It asks whether pair training on flowcharts improves layout invariance on unlabeled abstract graphs. No reported result was found on Lacuna for a trained vision-language model on those tasks, so the untuned base model is the reference.

Finally, we test the accuracy–faithfulness tension raised by Decoding on Graphs. There, enforcing valid paths raised 2Wikimultihop Hits@1 from 80.92% to 84.06% while cutting invalid triples from 16.67% to 0.00%. In the chart work, by contrast, enforcing faithfulness cost accuracy. If our exactly verifiable reward behaves like the constrained-decoding case rather than the preference-training case, unpaired accuracy should rise alongside pair consistency. Tracking both across training settles which regime holds for visual diagrams.