Carrying Visual Concepts Across Reasoning Hops

Some questions about images need several lookups in sequence. "Is the bird in this photo the same species as the one on the cover of that field guide, and where does it winter?" Current systems for these multi-hop questions treat each image as an opaque blob. They either describe it in words, which throws away the fine detail the question depends on, or they hand the whole image back to a large vision-language model at every step and hope it attends to the right place. This proposal tests a different kind of memory. A sparse autoencoder (a small network trained to rewrite a model's internal activations as a few active units out of tens of thousands, each tending to mean one thing) is run over the image patches of a frozen CLIP encoder. The few concepts that fire on the region that matters, together with where they fire, become a compact "concept ledger." That ledger is carried from hop to hop and used to retrieve, crop, and compare evidence.

The bet is that the hardest multi-hop failures are failures to keep a specific local visual fact alive across steps, more than failures of planning. The numbers are consistent with that reading. On the MMhops benchmark, a trained 7B tool-using agent reaches 51.35% on bridging questions but only 22.01% on comparison questions, which require holding the state of several images at once (MMhops-R1). If a ledger of localized, unnamed concepts closes part of that gap, and does so better than equally local but unstructured patch embeddings, it would show that unsupervised concept discovery is useful as working memory for reasoning, not only as an auditing tool for single images.

Core Claim

The core claim is as follows. Representing intermediate visual state as a sparse set of patch-localized autoencoder concepts, rather than as captions, whole-image embeddings, or attention maps, improves multi-hop multimodal question answering. The gain should be concentrated on comparison and image-bridged questions. It should come from two properties that the alternatives lack in combination: the state is local, so global scene traits do not contaminate it, and it is sparse and discrete, so it can be indexed and carried cheaply without being put into words.

The claim is deliberately falsifiable against its nearest cousin. If dense patch embeddings selected by the same localization step do equally well, then "local" is the ingredient and "concept" is not, and the paper must say so.

Four directions in the literature feed this claim, each in a different role. Concept discovery supplies the mechanism. The global-local gap supplies the reason a whole-image representation is the wrong state. Multi-hop reasoning is the application regime. The cross-modal gap is the constraint that rules out verbalizing the state.

What Prior Work Establishes and Leaves Open

The mechanism comes from patch-level sparse autoencoders on CLIP. PatchSAE trains an autoencoder on layer-11 patch activations of a frozen CLIP ViT, expanding them to roughly 49,000 sparse units. It shows that these units form spatial masks for concepts such as "dog noses," and that downstream adaptation remaps these detectors to labels rather than creating new ones. That stability is what makes a fixed, precomputed concept index plausible across tasks. ConceptScope shows that autoencoder heatmaps localize concepts better than VLM attention (AUPRC 0.399 against 0.098–0.302) and predict concepts better than captioning models (F1 0.72 against 0.58 for LLaVA-NeXT). Steering CLIP's vision transformer with sparse autoencoders shows that patch tokens keep dense information across layers while the global CLS token compresses it. It also documents a center bias, in which central patches activate more units, which this proposal must control for.

All three papers stop at single-image classification or dataset auditing. None asks whether their concepts can serve as memory for a reasoning process.

Two lines of work constrain the design. Learning Concept Bottleneck Models from Mechanistic Explanations shows that discovered concepts beat concepts chosen in advance by an LLM (90.04% against 62.03% Macro ROC-AUC on CUB). However, it names each unit by asking GPT-4o, and it reports that this naming step produces generic or hallucinated labels. Improving Concept Alignment in Vision-Language Concept Bottleneck Models identifies the "halo effect," in which global traits such as dominant color leak into fine attribute scores in CLIP's shared space. It also states that checking whether a concept score comes from the right image region remains open. Together these push the design toward concepts that are never required to be named and that are always tied to a location.

The reconstruction result in PatchSAE, 64.8% ImageNet accuracy from reconstructed activations against 68.2% from the originals, is a further constraint: autoencoder codes lose information. The ledger is therefore used only to decide where to look and what to retrieve. The answering model always sees real pixels.

The application regime is multimodal multi-hop QA. MMAPG replaces linear reasoning chains with a planning graph that can branch and backtrack, scoring 70.6 F1 on MultimodalQA and 65.9 accuracy on WebQA. It treats images at the whole-document level and its vision model as a black box. MMhops-R1 trains a Qwen2.5-VL-7B agent with image and text search tools. QA-Dragon chains routers and rerankers and reports that fine-grained discrimination of similar classes, such as plant species, stays weak. ChainMPQ is the closest in spirit: it carries attention-derived masks as "visual memory" between sub-questions and lifts LLaVA-1.5 on MMRel from 59.02% to 65.20%. Its masks are unverified polysemantic attention, however, and it works within one image without retrieval.

On the theory side, the Fano-style bound argues that single-pass accuracy collapses once the information a question demands exceeds what one pass can output. It also warns that compressing intermediate state to stay under that limit can drop crucial constraints. The ledger is an attempt at compression of visual state that keeps those constraints.

Research Program

The system has three parts, all built on a frozen CLIP ViT and an unchanged answering model.

The first part is an index. A PatchSAE-style autoencoder is trained on patch activations from the retrieval corpora: WebQA's 390k images and the MMhops image pool. Because each patch activates only a few dozen units, the corpus can be stored as an inverted index from concept to (image, patch, activation). This revives the "visual words" idea with learned, mostly monosemantic words.

The second part is the ledger. When a hop identifies a region of interest, the ledger records the top-activating concepts in that region and their weights. The region comes either from the question's own image or from a grounding sub-question in the style of ChainMPQ.

The third part is a pair of tools that use the ledger. Concept retrieval scores a candidate image $I$ against a ledger $L$ by

$$s(I) = \max_{p} \sum_{k \in L} w_k \, a_k(I, p),$$

where $p$ ranges over patches, $w_k$ is the ledger weight of concept $k$, and $a_k(I,p)$ is that concept's activation at patch $p$. This returns both a score and the best-matching location, and the location is used to crop the image before it reaches the answering model.

Concept comparison takes the ledgers of two localized regions, possibly from different images. It returns the concepts that are present in one and absent in the other, each with its heatmap, along with the corresponding crops. The answering model then sees matched local evidence rather than two whole images.

These tools are added to an existing planner rather than replacing it. The main configuration uses an MMAPG-style training-free planning graph. A second configuration gives the tools to the MMhops-R1 policy without retraining, to test whether the gain transfers.

Alongside the system, the program includes a failure analysis on a few hundred MMhops comparison errors made by the baselines. Each error is labeled by whether the decisive fact was a local visual attribute that was lost or misread, a retrieval miss, or a planning error. This labeling decides in advance how much the ledger can possibly help. If local-attribute loss accounts for few errors, the project's ceiling is low, and we will report that.

Baselines

The naive reference is the answering model alone, prompted to think step by step over the full images and questions with no retrieval structure. The file gives GPT-4o at 36.62% on MMhops bridging and 8.76% on comparison (MMhops-R1), and Gemini-2.5-flash at 46.58% on bridging. We will rerun Qwen2.5-VL-7B in this mode as our own floor.

The strongest existing methods are MMhops-R1 on MMhops (51.35% bridging, 22.01% comparison) and MMAPG on WebQA (65.9 QA-accuracy) and MultimodalQA (70.6 F1). For MultimodalQA cross-modal questions specifically, the file reports ImplicitDecomp at 51.7 F1 (MultiModalQA). We will rerun MMAPG with the same backbone, the same retrieval corpus, and the same cap on model calls per question, so that any gain cannot come from extra compute. The ledger version receives no additional tuning budget beyond selecting the number of ledger concepts on a validation split. MMhops-R1's reported numbers will be quoted, and its released policy rerun with and without the ledger tools.

Beating the strongest baseline means a gain on MMhops comparison over our reproduction of MMhops-R1 or MMAPG that holds across three seeds of question sampling, with a smaller or zero gain on text-only bridging. That pattern is what the mechanism predicts.

Three ablations isolate the key ingredient, each using the same planner, crops, and call budget. The first replaces the ledger with the raw dense patch embeddings of the same region and retrieves by cosine similarity, which tests whether sparsity and concept structure matter beyond locality. The second replaces it with the CLIP global embedding of the cropped region, which tests the patch-level structure itself. The third replaces it with ChainMPQ-style attention masks as the carried state. A fourth variant names every ledger concept with an MLLM, as in M-CBM, and carries the names as text. This measures what is lost when the concepts are verbalized.

Why not just…?

Why not just caption each image and run a mature text multi-hop pipeline? Captioning is the cheapest bridge across the modality gap, and it is exactly the loss this proposal targets. WebQA found that converting images to captions and tags left image-based queries far behind text-based ones (roughly 28% against 50%). ConceptScope shows captioning models describe concepts inconsistently. The named-concept ablation above is the direct test: if text names match the unnamed ledger, verbalization is not the bottleneck and the proposal's motivation weakens.

Why not just crop the relevant region and retrieve with ordinary CLIP or patch embeddings? This is the real threat. Locality may do most of the work, and the autoencoder may add nothing. The dense-patch and cropped-global ablations settle it. Two arguments favor the ledger. The halo effect documented by Improving Concept Alignment in Vision-Language Concept Bottleneck Models predicts that dense embeddings still carry background and tone into the match. Sparse codes also permit an inverted index and the set-difference comparison, which dense vectors do not give directly. If the ablations tie, we will reframe the contribution as local state carryover and drop the concept claim.

Why not just carry attention maps between steps, as ChainMPQ does? It is training-free and already works within one image. However, attention localizes worse than autoencoder heatmaps by ConceptScope's measurement, and an attention map says where the model looked, not what it saw there. That means it cannot serve as a key for retrieving other images. The attention-state ablation measures this on cross-image comparison, where we expect the gap to be largest.

Why not just train the agent harder with reinforcement learning? MMhops-R1 already does so and still reaches only 22.01% on comparison, which suggests the limit is not only policy quality. Plugging the ledger tools into its unchanged policy tests whether the two approaches are complementary.

Decisive Checks

Three results would decide the project.

The first is the main comparison on MMhops comparison questions against the dense-patch ablation at equal call budget. This is the experiment that justifies the word "concept."

The second is a faithfulness check. On a few hundred MMhops and WebQA questions, annotators mark the region that actually contains the decisive evidence. We then measure how often the ledger's retrieved location falls inside that region, compared with attention-based and global-embedding retrieval. A center-only heatmap serves as a control for the ViT center bias reported in Steering CLIP's vision transformer with sparse autoencoders.

The third is a halo test. On WebQA image queries, we pair questions whose images differ in global tone but share the queried local attribute, and check whether ledger retrieval is more stable than global-embedding retrieval.

We will also report context size per hop. A ledger of a few dozen concept indices plus one crop should cost far fewer tokens than full images. That would give an empirical handle on the Fano-style argument that compact state prevents single-pass collapse.

No reported result was found on Lacuna for any concept-based or autoencoder-based method on MMhops, WebQA, or MultimodalQA, so every ledger number in this study will be new.