Multimodal models often see an image correctly and still reason about it wrongly. On competition math problems with diagrams, most of GPT-4V's errors come after every visual element was recognized correctly, not from misreading the picture Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset. A separate line of language research has found that knowledge systems built mostly on "is-a" hierarchies (a car is a vehicle) break down when the task needs other relations: part-whole (a car has wheels), material (a car is made of steel), and function (a car is used for driving). We propose that these two findings describe one failure. Different relation types permit different inferences. "Is-a" passes properties down: if dogs are animals and animals breathe, dogs breathe. "Part-of" does not: a wheel is part of a car, a car is a vehicle, and a wheel is still not a vehicle. Our hypothesis is that vision-language models apply is-a-style inference to relations that do not support it, and that much of the "reasoning error" category is this one rule being misapplied.
The project has two parts. The first is a diagnostic benchmark on real photographs. Each question pairs a perception step (identify the part, object, or material in the image) with one relational inference step. The relation type varies while perception difficulty is held fixed, and items where the is-a inference would be valid are matched with items where it would be invalid. The second is a light intervention. The model writes its reasoning as typed relation steps, and a small rule checker rejects steps that use an inference the relation type does not allow. The central test separates two explanations. If supplying the missing relational facts fixes the errors, the problem is missing knowledge. If the errors persist until the relation-specific inference rules are enforced, the problem is rule misuse, and only the second result would make this work new.
Core Claim
Holding perception fixed, vision-language models will show a measurable inheritance-violation rate: the fraction of items on which they wrongly carry a property or category across a part-whole, material, or function link. This rate will be much higher than their error on matched is-a items where the same inheritance is valid. Giving the models the relevant facts in text will reduce their errors on missing-knowledge items but will barely change this rate. Enforcing relation-specific inference rules at each reasoning step will reduce it.
This differs from the closest work in a specific way. Scene-graph training, as in Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models, builds graphs of objects, sub-parts, and spatial or interaction edges and trains models to reason over them. It treats edges as generic links, not as relations with different inference rules, and it is evaluated on spatial perception. The part-based benchmark PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning brings part-whole structure into visual reasoning, but it uses synthetic furniture and template programs. It does not ask whether models confuse one relation type with another. Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning" separates perception from logic with differentiable first-order logic, but only over GQA's attribute and spatial predicates. None of these varies the relation type while holding perception constant, which is the measurement this project depends on.
Grounding in the relation-coverage literature
Direction 1 supplies two things: evidence that relation types are not interchangeable, and a warning about lexical resources. On the evidence side, cross-lingual similarity measured over HowNet rose from 0.8594 to 0.9035 Pearson correlation only when part-whole, material-product, and attribute links were added to tree distance An English-Chinese Cross-lingual Word Semantic Similarity Measure Exploring Attributes and Relations. Graded entailment needed an explicitly asymmetric network, because symmetric embeddings cannot represent "is-a" direction Scoring Lexical Entailment with a Supervised Directional Similarity Network. That paper also handled only one relation type, which is the gap our multi-relation checker targets.
The warning comes from formal auditing. When WordNet's part-whole pairs were checked against the SUMO upper ontology with automated theorem provers, 14,513 of 22,187 pairs (65%) hit type-restriction errors Towards Cross-checking WordNet and SUMO Using Meronymy. Lexical part-whole data is therefore too noisy to use directly as gold labels. We reuse that paper's checking procedure as a first filter, then send what survives to human validation. Both Contextual Data Cleaning with Ontology Functional Dependencies and the taxonomy-mapping study A Large Scale Taxonomy Mapping Evaluation lead to the same design lesson: systems that look good on clean synthetic structure lose much of their performance on real data. In the mapping study, recall fell from 70–80% to 30–40%. This is why we use real photographs and do not extend PTR's renders.
Research Program
The benchmark. We will build a relation-type diagnostic set from real photographs, using COCO and SA-1B images of the kind SaGe draws on. Each item has three layers. The first is a perception probe, such as "what is the object circled in red?" or "what is this handle made of?". The second is a one-step relational question, which conditions on the probed entity and uses a WordNet or HowNet link that passed the SUMO check and human validation. The third is a text-only twin, in which the perceived entity is stated in words.
Relation type takes four values: hypernymy, meronymy, material, and function. For each type we build valid-inheritance items and invalid-inheritance items. A valid hypernymy item asks whether a dog in the image breathes. An invalid meronymy item asks whether the wheel shown is a vehicle. An invalid material item asks whether a steel spoon is a metal. An invalid function item takes a knife, which is used for cutting, and asks whether it is used for whatever cutting is used for.
Items are scored only when the model passes the perception probe. This follows the logic of the error split in MATH-Vision, but makes it automatic and applies it per item. The headline measure is the inheritance-violation rate on invalid items minus the error rate on matched valid hypernymy items.
The intervention. The model is prompted, or lightly fine-tuned, to write each reasoning step as a typed triple, for example (wheel, part-of, car). A rule table, written once from standard ontology axioms, states which inferences each relation type allows. "Is-a" is transitive and passes properties down. "Part-of" composes only within the same part subtype and passes no category membership. "Made-of" passes material properties but not category. "Used-for" is not transitive. A step that breaks the table is rejected and resampled, up to a fixed budget.
The rule table is deliberately small and fixed. We do not expect it to fix perception, and it cannot repair facts missing from the knowledge source. That limitation is exactly what makes the fact-versus-rule ablation informative.
Transfer. We then ask whether the checker helps on existing benchmarks where these relations come up naturally. The two targets are PTR's part-based and physics questions and the subset of ScienceQA questions about materials and functions. We also confirm that general multimodal ability does not drop.
Baselines
The naive reference point is the base model answering directly, with no reasoning prompt. We will run Qwen2.5-VL-7B, LLaVA-1.5-13B, and GPT-4o on every condition. These models are chosen because the cited papers report them: Qwen2.5-VL is SaGe's base, LLaVA-1.5-13B is DCoT's base, and GPT-4o appears in MATH-Vision. On our new benchmark this baseline sets the inheritance-violation rate everything else is measured against. There is no reported result found on Lacuna for any model on a relation-type-controlled visual inference set.
The second baseline is free-form chain-of-thought (step-by-step reasoning written in prose), together with self-consistency voting under the same sample budget as the checker's resampling. It is the natural competitor because it is free, and it is also known to be unstable on multimodal tasks. MATH-Vision reports that it lowers accuracy for some models. On that benchmark GPT-4o reaches 30.39% against 68.82% for humans Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset. On ScienceQA, GPT-3 with chain-of-thought reaches 75.17% Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.
The third baseline is retrieval of the relevant facts. For every item, the exact WordNet or HowNet triples involved are inserted into the prompt. This follows the spirit of DCoT, which pairs visual box hints with retrieved demonstrations and reports 38.4% versus 36.1% on MM-Vet with LLaVA-1.5-13B DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models. This is the most important baseline for our claim, because it gives the model every fact while giving it no rules.
The strongest trained competitor is SaGe-style training. Following Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models, Qwen2.5-VL-7B is fine-tuned on scene-graph reasoning data. We add typed relation edges from our training split, using the same number of examples and the same compute as the checker's fine-tuned variant. SaGe-7B reports 79.4 on CVBench-2D, 80.5 on CVBench-3D, and 62.3% on MMStar. We use the MMStar figure as the bar for checking that general ability is not lost.
The key ablation keeps the typed-triple output format but disables the rule table, so that every step is accepted. This separates the effect of structured writing from the effect of rule enforcement.
On PTR, the strongest reported system is NS-VQA, at 62.1% overall and 41.2% on Physics, against 84.8% for humans PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning. The criterion for beating the baselines has two parts. On the diagnostic set, the checker must cut the inheritance-violation rate by a margin larger than the across-seed confidence interval, compared with both fact retrieval and the format-only ablation, without raising errors on valid items. On transfer, it must improve the meronymy-heavy PTR and ScienceQA subsets over chain-of-thought while matching the base model on MMStar.
Why not just…?
Why not just give the model the facts? If the failure is missing knowledge, retrieval is the cheaper fix and should win. Our hypothesis predicts something narrower: retrieval lowers errors on valid items and on "does it have this part?" items, but leaves the violation rate on invalid-inheritance items largely unchanged, because the model holds the fact and still draws the wrong inference. The retrieval baseline and the checker are both run on every item, so this prediction can be checked directly. If retrieval closes the gap, we will report that the gap is a knowledge gap, and the rule checker will not be needed.
Why not just use a better vision encoder or a gold caption? The perception probe already conditions on correct perception, and the text-only twin removes the image entirely. GeoSQA shows that even complete human transcriptions of diagrams leave text models near chance, with BERT at 23–25% against 25% random GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level. If the violation rate falls to the level on valid items once perception is perfect, the encoder explanation wins, and our setup will show that.
Why not just fine-tune on typed scene-graph data, as SaGe does? This is the most serious threat. Training on typed relation data could teach the inference rules implicitly. Results from PTR, where neural models degrade sharply with less data, and from ScienceQA, where explanation-trained models matched baseline accuracy with 40% of the data, suggest that explicit structure helps with data efficiency. Neither result shows that it is required.
We settle this by testing on held-out relation pairs and held-out object categories that are absent from the fine-tuning data. If SaGe-style fine-tuning matches the checker there, we will report that rule enforcement can be learned and that the checker is a cheaper, training-free version of the same fix. The diagnostic finding stands either way.
Why not just prompt the model to "consider the relation type"? This is cheap, and we include it as a prompt-only variant of the format ablation. If naming the relation is enough without any rejection step, the contribution reduces to the diagnosis, which is still worth reporting.
Decisive Checks
Three results would confirm the claim, and they must hold on all three models. First, the inheritance-violation rate on part-whole, material, and function items must exceed the error on matched valid "is-a" items, conditional on correct perception and also in the text-only twin. Second, fact retrieval must reduce errors on knowledge-gap items while leaving the violation rate within noise. Third, rule enforcement must reduce the violation rate where format alone does not.
The result that would falsify the claim is a violation rate that does not differ from valid-item error. That would mean the models' reasoning failures are not organized by relation type, and the coverage-gap framing would not carry over to multimodal reasoning.
A secondary check tests whether the ontology filter matters. We compare items built from the human-validated subset of SUMO-consistent pairs with items built from raw WordNet pairs. Given the 65% conflict rate, a large gap between the two would show that earlier probing work on lexical part-whole data was partly measuring label noise.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
- PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
- Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"
- An English-Chinese Cross-lingual Word Semantic Similarity Measure Exploring Attributes and Relations
- Scoring Lexical Entailment with a Supervised Directional Similarity Network
- Towards Cross-checking WordNet and SUMO Using Meronymy
- Contextual Data Cleaning with Ontology Functional Dependencies
- A Large Scale Taxonomy Mapping Evaluation
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level