A web caption such as "a crowd outside a stadium" does the same thing a scalable logic engine does on purpose. It collapses many individuals into one summary object and records only what they share. Work on caption sparsity treats this collapse as a defect and fixes it by describing everything densely, either once at training time or once per image at inference time. That approach is expensive, and every extra generated sentence is another chance to hallucinate. Work on scaling symbolic reasoning takes the opposite view. The SHER system for large ontologies (Scalable Semantic Retrieval through Summarization and Refinement) deliberately merges individuals that share the same properties into summary nodes. It runs the logic check on this small summary and splits a summary node only when the proof of an answer actually depends on it. We propose to carry that discipline into vision. A reasoner answers a question over a coarse, conservative scene description. When the answer cannot be decided, it identifies the specific group-level facts responsible. Only the image regions behind those facts are then re-captioned in detail.
The core claim is that the logic engine's own record of which facts a conclusion rests on (its justification) is a better guide for where to spend perception effort than either "describe everything" or "let the model pick a crop." If the claim holds, a visual question-answering system should match the accuracy of full dense scene description while making several times fewer calls to the captioner. It should also generate far fewer unverified assertions, and it should do better than heuristic zooming on questions about counting and spatial relations. The new element is that refining the symbolic abstraction triggers new perception. In SHER, refinement only re-partitions facts that already exist. Here, it produces new visual evidence.
Why this intersection, and what is already known
The case for density comes from a consistent finding. Sparse text lets models ignore secondary objects, counts and layout. Recaptioning a billion images helps, but only when about 80% raw web text is mixed back in, because purely synthetic captions hurt zero-shot ImageNet and bring in hallucinations (What If We Recaption Billions of Web Images with LLaMA-3?). SEPS (SEPS) reports that injecting generated dense text causes semantic drift unless it is calibrated. Structured descriptions beat free text: Visual Tables, which are JSON-like per-object attribute records, outperform scene graphs by 15.4 points on MMVP in text-only mode (Beyond Embeddings: The Promise of Visual Table in Visual Reasoning). These papers motivate us. Density matters, but uniform density is costly and noisy. Visual Tables are still handed to a language model as a prompt, and nothing checks them formally.
The case for a formal reasoner comes from two results. NSVQASP (A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering) pairs a detector with Answer Set Programming, a declarative logic language whose solver searches for consistent models. It reaches 98.22% on CLEVR-NEW against 80.76% for relational networks. Its authors state that it does not extend to real images, because noisy detectors and large attribute spaces blow up the search. Find The Gap shows that language models given the correct facts still fail at two-hop composition. These papers constrain us: a real-image symbolic system needs some way to keep the fact base small and trustworthy. SHER supplies the method. It uses conservative summaries, sound pruning (if the summary is consistent with the negation of the query, every member is a guaranteed non-answer), and refinement driven by justifications. On the UOBM-10 ontology benchmark it answered queries in 15.4 seconds where KAON2 took 447.6. SHER only ever operated on text knowledge bases whose facts were fixed in advance.
The closest perception-side work is heuristic. Zoom-Refine (Zoom-Refine) has the model propose a bounding box and re-read a high-resolution crop, and it notes that bad localization ruins the refinement. Draft and Refine (Draft and Refine with Visual Experts) invokes visual experts when a masking-based utilization score is low. LASER (LASER) turns sparse captions into temporal logic and trains through a differentiable reasoner, but it takes caption density as given. Its exact solver also limits it to short, small scenes. None of these lets a proof decide what to look at next.
Core Claim
We make the claim precise with a conservative summary scene graph. A group node $g$ stands for a set of image instances $M(g)$ of unknown or bounded size, with a count interval $[\ell_g, u_g]$ and a region. It carries only the attributes and relations the coarse describer asserts hold for all members. The question is compiled into a logic program. The solver then assigns one of three outcomes: decided-true, decided-false, or undecided. An outcome is decided when every way of filling in the members of each group, consistent with the group's shared facts and count interval, gives the same answer. Undecided cases come with a minimal justification set of group nodes. Refinement runs a targeted describer and detector on only those regions, splits the groups into instances or smaller groups, and loops until the answer is decided or a call budget $B$ is spent. If the coarse assertions are correct, answers decided at the summary level agree with the answers on the full dense graph. We test this directly with an oracle, as described below. The empirical bet is that most questions are decided after refining a small fraction of the scene.
Research Program
We would build four components. The first is a coarse describer: an off-the-shelf multimodal model prompted to emit a summary graph in a fixed schema, with group nodes, count intervals and shared attributes. The second is a question-to-program translator in the style of NSVQASP, targeting a small Answer Set Programming fragment that supports counting, negation and spatial relations. The third is a justification extractor that returns the minimal group nodes whose splitting could flip an undecided answer. We adapt SHER's minimal-inconsistent-set procedure to three-valued outcomes. The fourth is a refinement operator that crops the justified regions and runs the same describer plus an open-vocabulary detector, the Grounding DINO and SAM pipeline used to build the Patch-Aligned Dataset (Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models).
The analysis half of the program asks three questions. How often is the summary alone decisive? How does accuracy grow with the refinement budget? How many generated assertions does each method expose the reasoner to? That third quantity stands in for hallucination risk, since an assertion that is never generated cannot be hallucinated. Where Visual Genome scene graphs exist, we also measure how many generated assertions are actually wrong.
Baselines
The naive reference point is the same multimodal model answering the question directly from the image with chain-of-thought, with no scene structure. It shows whether any of the machinery helps.
The strongest competitor on accuracy is full dense description followed by reasoning. The Visual Table recipe describes the whole image up front and feeds the table to the language model; the file reports it adds 4.4 points on MM-Vet and 3.4 on MMVP over LLaVA-1.5 (Visual Table). We run two versions with the same describer and detector as our system. In one, a language model reasons over the dense table. In the other, our own solver runs over the fully dense graph. The second version isolates the value of summarization from the value of symbolic reasoning. We beat this baseline if we match its accuracy within one point at a third or less of its describer calls, or exceed it on counting and spatial subsets.
The strongest competitor on targeted perception is Zoom-Refine. On HR-Bench 8K it reports a gain of 5.0% average accuracy for InternVL3-78B, and 4.2% on HR-Bench 4K. The file gives only gains, and no reported absolute result was found on Lacuna for HR-Bench. We run it with the same backbone and give it the same number of crops as our budget $B$. Beating it means higher accuracy at equal crop count on HR-Bench and MME-RealWorld, with the gap concentrated in multi-object questions.
On synthetic scenes, the symbolic reference is NSVQASP at 98.22% on CLEVR-NEW. There we expect to match it while showing lower solver time on scenes we inflate with many duplicate objects. On knowledge-dependent questions, the reference is the retrieval network from Find The Gap at 44.36% on KRVQA.
The key ablation keeps the loop and the budget fixed and changes only how refinement targets are chosen. The options are justification sets, uniformly random group nodes, the largest group nodes, and the model's own proposed box. If justification-guided choice does not beat these at equal budget, the core claim fails. All systems use one backbone, identical prompts where applicable, and the same tuning budget of a held-out 500-question development split per benchmark.
Why not just…?
Why not just describe everything densely and let the language model reason? This is the most serious threat. At equal accuracy, it is simpler. The evidence against it as a complete answer is indirect: recaptioning papers show that dense synthetic text carries hallucinations, and Find The Gap shows language models fail at two-hop composition even when given correct facts. Neither result directly shows that on-demand densification wins. Our first experiment settles this by plotting accuracy against describer calls for both approaches. If the dense-table curve dominates, we will report that and conclude that justification guidance buys nothing beyond cost.
Why not just let the model choose where to zoom, as Zoom-Refine does? A self-chosen crop works when a single region holds the answer. Counting and comparison questions need evidence spread over several places, and Zoom-Refine itself reports that wrong boxes break the refinement. The equal-crop comparison on multi-object subsets tests whether justification sets localize better.
Why not just filter hallucinated captions with a detector and skip the reasoner? Detector verification removes false object claims. It does not tell you which unmentioned facts the question needs, and it does nothing for relations or counts that no single detection confirms. We include a "dense description plus detector filtering" variant inside the dense baseline, so this comes out of the measurements rather than being argued.
Why not just run the solver on the full dense graph, since one image is small? For a typical photo this may well be tractable, and we say so. On such images our advantage should appear in perception cost, not solver time. Solver-time gains should appear only on 4K and 8K images and on inflated CLEVR scenes. The dense-graph-plus-solver baseline measures exactly where that boundary lies.
Proposed Experiments
The first decisive check isolates the logic from perception. On GQA questions whose images have Visual Genome scene graphs, we build summary graphs from the ground-truth graphs themselves, merging instances with identical shared attributes. We verify that every answer decided at the summary level matches the answer on the full graph. Any disagreement means the three-valued semantics or the justification extractor is wrong. The same check reports the fraction of questions decided with zero refinements. If that fraction is near zero, the savings cannot exist and the project should stop there.
The second check uses real perception on GQA, CLEVR-NEW, HR-Bench 4K and 8K, and MME-RealWorld. We plot accuracy against describer calls and against generated assertions for our system, the dense baselines, Zoom-Refine and the ablations. We break the results down by question type, separating counting, spatial relations, negation and attribute questions. The prediction is that our curve reaches the dense baselines' accuracy at a third or less of the calls, and that the advantage over Zoom-Refine is largest on counting and spatial questions.
The third check applies the loop to knowledge questions on KRVQA. Refinement may then target either image regions or knowledge-base lookups. This tests whether one justification mechanism can decide between looking again and retrieving, which is the static-pipeline gap Find The Gap leaves open.
References
- Scalable Semantic Retrieval through Summarization and Refinement
- What If We Recaption Billions of Web Images with LLaMA-3?
- SEPS: Semantic-Enhanced Patch Slimming Framework for Fine-Grained Cross-Modal Alignment
- Beyond Embeddings: The Promise of Visual Table in Visual Reasoning
- A Logic-based Approach to Contrastive Explainability for Neurosymbolic Visual Question Answering
- Find The Gap: Knowledge Base Reasoning For Visual Question Answering
- Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement
- Draft and Refine with Visual Experts
- LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models