Grounding Before Deducing: Small Early-Fusion Binders for Multimodal Logic

When vision-language models fail at formal logic problems posed over images, they rarely fail at the logic itself. On MuSLR-Bench, a set of 1,093 problems that pair images with propositional, first-order, and non-monotonic rules, about 70% of GPT-4.1's errors are "logical misalignment": the model sees the right objects and knows the right rules, but attaches what it sees to the wrong logical variable (MuSLR). Current fixes respond with more language-model calls. The paper's own LogiCAM loop runs premise selection, reasoning-type identification, reasoning, and a completion check as repeated prompts. The ARC-AGI-2 pipeline of Compositional Neuro-Symbolic Reasoning chains three frontier models and admits a "multiplicative inference cost." Nobody has measured how much energy any of this uses.

We propose to treat the hard part as a separate, small, checkable step. A compact model reads image and premise text together from its first layer, which is what "early fusion" means. It emits only one thing: a short program of grounding operations. Each operation states whether an atomic fact holds in the image and points to the evidence for it. A deterministic solver then does all the deduction. In two sentences: assignment of truth values from pixels to logical atoms becomes an explicit, auditable operation sequence produced in one cheap forward pass, and multi-step reasoning moves to a solver that costs almost nothing and does not get worse with depth. If this works, it should raise accuracy where misalignment dominates, give a hard grounding trace in place of attention heatmaps, and cut energy per correct answer by an order of magnitude compared with iterative prompting loops.

Core Claim

Our claim is that logical misalignment is an output-space problem, not a capacity problem. Models misalign because they are asked to ground and deduce in the same free-form text stream. Prior work on interpretability shows what happens when a neural model must instead emit explicit operations.

An Operation Sequence Model for Explainable NMT recasts translation as grammar operations such as source pops, markers, inserts, and jumps. This cut Japanese-English alignment error rate from 54.7% to 21.5% with no loss in BLEU. EditNTS does something similar for simplification, emitting keep, delete, and add actions that a parameter-free interpreter executes. Both papers supply the method we reuse: a constrained operation vocabulary executed by a deterministic interpreter. Both are text-only. We carry the idea to the step where multimodal input is bound to symbols.

The binder's vocabulary is small. It has an operation that asserts a predicate over named constants as true, false, or unknown, plus a pointer to a set of image patches or premise-text spans as evidence. It has an operation that introduces a constant for a visual entity, and a stop operation. The solver receives the resulting fact set together with the symbolic premises. It is a propositional/first-order prover, or an answer-set solver for the non-monotonic items.

Early fusion is the right regime for the binder, and the evidence is specific. FuseLIP tokenizes images into discrete tokens and feeds them with text into one encoder. On the OI-Pos spatial benchmark it reaches 68.9% against roughly 48% for late-fusion models. Grounding a predicate like "the pedestrian is left of the car" is exactly this kind of spatial, text-conditioned judgment. AgentRx likewise finds that one model ingesting all modalities beats multi-agent debate. These results motivate the design. The constraint from FuseLIP is that early fusion grows sequence length quadratically. That is tolerable here because the binder is small and runs once.

Research Program

The first component is the binder. It is an encoder-decoder of a few hundred million parameters. The encoder takes discrete image tokens concatenated with the premise text, following FuseLIP's single-encoder design. We will also try a patch-token variant so that evidence pointers land on spatially meaningful units. The decoder emits the operation sequence under a grammar mask, so every output can be parsed. One known weakness carries over from operation-sequence models: different operation orders can produce the same fact set. We remove it by canonicalizing order (constants first, then atoms in lexicographic order) and by training against the canonical form.

The second component is training data. MuSLR-Bench is too small to train on and must stay held out. MuSLR builds its items from COCO and Flickr30k imagery plus templated logic, so we will procedurally generate binding supervision the same way. Object and relation annotations on COCO images become ground-truth atom truth values. Templated rule sets are then written over those atoms. This yields hundreds of thousands of image–premise–fact-set triples. We will deduplicate them against every image used in MuSLR-Bench.

The third component is the solver interface. Every problem becomes a check of whether the fact set plus premises entails the queried conclusion, its negation, or neither. Unknown atoms are passed to the solver as unknown rather than forced to a value.

The fourth component is measurement. All open-weight systems run on the same GPU. Energy is logged with the ML-EcoLyzer methodology at 5 Hz, because that paper reports that 1 Hz sampling overestimates CO₂ by nearly 6%. We report joules per item and joules per correct answer, and we separate the binder's energy from the solver's.

Baselines

The naive reference point is GPT-4.1 with standard chain-of-thought on MuSLR-Bench, reported at 46.8% accuracy (MuSLR).

The strongest existing method is LogiCAM on GPT-4.1. The paper reports that it "boosted GPT-4.1's performance by 14.13% overall" and beat the base model by 13% at reasoning depths 8–9. We quote these figures as reported rather than converting them, since the paper's phrasing does not settle whether the gain is absolute. No reported result found on Lacuna for the energy or latency of LogiCAM or of any MuSLR-Bench system.

Because proprietary APIs cannot be metered, the energy comparison has to use open weights. We will reimplement LogiCAM's four modules on InternVL, which MuSLR reports at 45.20% (above GPT-4o's 38.93%). We will also run its INT8 quantization, which ML-EcoLyzer reports saves 25–55% power with above 94% accuracy retention. For GPT-4.1 LogiCAM we report accuracy and count calls and tokens as a cost proxy. Our method beats LogiCAM if it matches or exceeds open-weight LogiCAM accuracy at a tenth of its joules per correct answer or less. It must also stay within a few points of the reported GPT-4.1 LogiCAM accuracy.

A third baseline isolates the solver from the trained binder. We prompt InternVL to write the facts and premises directly as solver-ready logic and send them to the same solver. This is the "prompted autoformalization" competitor.

Two ablations isolate our key ingredients. A late-fusion binder captions the image first and then binds from text alone; this tests whether early fusion matters. A free-form binder has the same model and data but emits unconstrained logic strings instead of the grammar-masked operation sequence; this tests whether the operation grammar matters.

All trained variants get the same synthetic data, the same number of training steps, and the same hyperparameter search budget. All prompted variants get the same few-shot examples.

Why not just…?

Why not just quantize LogiCAM and run it on a small open model? That cuts energy per call, but not the number of calls. It also leaves the grounding error where it was, because grounding is still done in free text. MuSLR reports that 11–18% of open-model errors are basic perception failures on top of misalignment. We expect quantized LogiCAM to keep its error profile while trimming energy by a constant factor. The open-weight LogiCAM comparison measures exactly this.

Why not just prompt a VLM to write formal logic and hand it to a solver? This is the real threat. If prompted autoformalization on InternVL matches our binder's accuracy at similar energy, the trained binder is unnecessary and we will say so. We expect it to fall short for two reasons. Free-form generation leaves the misalignment mechanism intact: the model can still attach an observation to the wrong constant. And a large VLM decoding long logic strings costs far more than a small binder. Comparing the binding error rate of both systems on MuSLR items will show directly which one misaligns less.

Why not just add early exits or routing to chain-of-thought, as EcoThink does? EcoThink cuts text-only inference energy by 40.4% on average by sending easy queries to a small quantized model, and it notes that it does not handle multimodal input. Routing shortens reasoning without making it correct. On deep MuSLR items, a router would have to send most queries down the expensive path, so we expect small savings there. We will implement an EcoThink-style router over InternVL chain-of-thought and plot accuracy against joules by reasoning depth.

Why not just read cross-modal attention maps for interpretability? The operation-sequence work shows attention alignments are erratic (54.7% alignment error versus 21.5% for explicit operations). Our evidence pointers can be scored against gold atom regions from the synthetic data, and attention-derived pointers from the baseline models can be scored the same way.

Proposed Experiments and Decisive Checks

The main experiment reports MuSLR-Bench accuracy overall, by logic type, and by reasoning depth 2–9. It also reports joules per correct answer for every open-weight system. On a held-out synthetic split, we report binding error rate, meaning the fraction of atoms assigned the wrong truth value, and pointer accuracy against gold regions.

The first decisive check is depth invariance. If errors are really binding errors, our system's accuracy should be nearly flat across depths 2 through 9, because the solver does not degrade with depth. Prompted baselines should decline with depth. If our curve also declines, the binder is making compounding errors and the core claim is wrong.

The second check is error attribution. We hand-label 200 MuSLR failures from each system as perception, misalignment, or deduction errors. Our system should show near-zero deduction errors and fewer misalignments than LogiCAM. Any remaining errors should concentrate in perception, which would identify the image tokenizer as the ceiling, as FuseLIP's tokenizer-dependence warns.

The third check is the cost frontier. We plot accuracy against joules per correct answer for all open-weight systems. The proposal succeeds only if the binder-plus-solver system sits strictly above that frontier. It fails if prompted autoformalization or quantized LogiCAM reaches the same point.