Vision-language models still fail badly on competition math, scientific charts, and diagram-based exam questions. On MATH-Vision, GPT-4o scores 30.39% against a human average of 68.82%. A natural remedy is to give a model a growing external memory of problems it has already solved correctly and let it consult similar past solutions. Retrieval from a single large pool has a known weakness, though: it returns items that look alike (same chart type, same words in the question) rather than items that are solved the same way. We propose a different arrangement. Several identical, frozen copies of one model each keep their own private memory and may write into it only when they solve a problem correctly. A simple bidding rule decides which copy answers each new problem. A copy that happens to solve a few geometry-transformation problems early stores those solutions, wins more problems of that kind, and so stores still more. Specialization then comes from what each agent remembers, with no hand-designed roles and no weight updates.
The central claim is testable. At equal total memory size, a memory split this way among self-specializing agents should improve accuracy faster as memory grows than one pooled memory with an equally strong retriever. This advantage should appear only beyond a crossover size, the point where pooled retrieval becomes noisy. That prediction comes from multi-agent learning theory, which says agents need to differ only when each one has limited information. A pooled store that is too large to search precisely acts like limited information. A split store makes the limit deliberate and lets it be shaped by competence.
Core Claim and Where It Comes From
Three lines of evidence motivate the idea. First, the multimodal reasoning gap is largely a problem of reasoning, not only of seeing. In MATH-Vision, 42.2% of GPT-4V's errors were reasoning errors made after the visual elements had been read correctly, against 31.9% recognition errors. Chain-of-thought prompting even reduced accuracy for several models. GeoSQA replaced map diagrams with complete human-written descriptions, and neural models still scored near chance (23–26.22% on four-way questions). In that study, naively prepending retrieved textbook passages often hurt. The missing ingredient therefore looks like procedural knowledge of how this kind of problem is solved, and topical text does not supply it.
Second, external memory can substitute for scale when retrieval and reasoning are tied together. kNN-LM mixes a nearest-neighbor lookup over cached contexts into a language model's predictions. Fusion-in-Decoder encodes each retrieved passage separately and fuses them in the decoder; it kept improving up to 100 passages and let a 770M model beat an 11B closed-book model. IRCoT alternates reasoning steps with retrieval, which raised recall by 11–21 points. All three show that memory scales well for text facts. None addresses multimodal procedural memory, and all three use a single pooled store.
Third, theory on when specialization pays off sets the conditions under which our idea should work. When Is Diversity Rewarded in Cooperative Multi-Agent Learning? proves that the benefit of agents behaving differently, the "heterogeneity gain", depends on how team rewards are aggregated. It also proves that this benefit vanishes when agents have rich, complete observations. We take that result as a prediction rather than a method. Splitting memory should help only in the regime where a pooled memory no longer gives each query clean access to the relevant experience.
The closest systems connect roles, memory, and multimodal reasoning, but each designs its roles by hand. MATA trains a controller to move between three fixed sub-agents over an append-only shared log. PixelCraft prompts a grounder, planner, critics, and tool agents, and keeps an image "whiteboard" for backtracking within one problem. DART uses disagreement among different vision-language models to call in vision tools. Yu et al. combine debaters with a per-query data pool. In all four, memory lasts only for a single query, and roles are fixed before deployment. Our proposal does the reverse. Agents are interchangeable at the start, memory persists and grows across thousands of problems, and roles are defined by which memory each agent ends up holding.
Mechanism
We start from $N$ copies of one frozen open vision-language model, such as a Qwen2.5-VL-class model of the kind used in MATA and PixelCraft. Each copy owns a memory $M_i$. Each memory entry is a verified solution trace with three parts: the image and question, a symbolic extraction of the visual content (a table, coordinates, or a list of geometric relations, following the image-to-symbols approach of Chart-based Reasoning), and an executable solution program.
When a query $q$ arrives, every agent retrieves its top-$k$ neighbors and submits a bid:
$$b_i(q) = \frac{1}{k}\sum_{m \in \mathrm{NN}_k(q, M_i)} \mathrm{sim}(q, m)\,\mathbb{1}[m \text{ was reused successfully}]$$
Here $\mathrm{sim}$ is cosine similarity in a joint image-text embedding. The indicator equals 1 for entries whose past reuse as an exemplar led to a correct answer, so bids reward memories that have proved useful, not memories that merely look similar. The highest bidder answers, using its retrieved traces as worked examples and executing its program. During the experience stream, ground-truth labels determine correctness. If the answer is correct, the trace is written into that agent's memory only. Ties and empty memories at the start are broken at random, which supplies the initial asymmetry. This mirrors the observation in Deka et al. that small local asymmetries are enough to differentiate agents that share parameters.
The bidding draws loosely on the pricing coordination in Market Incentivization for Theorem Proving. We deliberately use no learned controller, because MATA reports that its controller's training cost grows quickly with the number of agents.
We define the heterogeneity gain at total memory size $S$ as
$$\Delta(S) = \mathrm{Acc}_{\text{split}}(S) - \mathrm{Acc}_{\text{pooled}}(S)$$
where both systems hold the same $S$ verified traces in total, and the pooled system has one memory containing all of them. The theory predicts that $\Delta(S)$ is near zero or negative for small $S$, where pooled retrieval is precise and splitting only fragments the evidence. It also predicts a positive and growing $\Delta(S)$ once pooled top-$k$ precision falls.
Research Program
The experience stream is built from the training portions of ChartQA (32.7K questions), VeriSciQA (20,272 pairs across 20 scientific domains), and MathVista. These are presented in a fixed random order. Memories are frozen at checkpoints of roughly 1K, 4K, 16K, and the full stream of verified traces. Evaluation is held out, uses no labels, and covers four sets. MATH-Vision is the hardest reasoning test. CharXiv allows comparison with PixelCraft. GAOKAO-MM tests transfer to exam-style physics and math diagrams. Chart101 from Plot Twist is run with neutralized titles, to check whether memory-based answering resists the title shortcut that pushed accuracy as high as 79% when titles revealed the answer.
We vary $N$ over 1, 4, 8, and 16. With $N = 1$ the system reduces to the pooled agent.
The analysis asks what the emergent roles actually are. We measure normalized mutual information between agent assignment and two candidate labelings. One is the subject taxonomy (MATH-Vision's 16 subjects). The other is a failure-type labeling: whether the zero-shot backbone fails a problem through misreading or through faulty reasoning, judged by checking whether its symbolic extraction matches the gold extraction. If roles track failure type more than subject, specialization reflects how problems are solved, which a topic-based split cannot reproduce.
Baselines
The naive reference point is the frozen backbone with zero-shot chain-of-thought and no memory. The closest reported analogue is PixelCraft's linear chain-of-thought baseline, 63.8% on CharXiv with GPT-4.1-mini. MATH-Vision's GPT-4o score of 30.39% gives the frontier ceiling for a single model. On MathVista, GPT-4V reported 49.9%; on GAOKAO-MM it reported 48.1%.
The strongest competitor on this specific question is a single agent with the pooled memory and the best retrieval we can build for it. It retrieves IRCoT-style, re-querying the memory after each reasoning step, and it reranks candidates by the same historical-success signal used in our bids. The hybrid episodic–procedural memory agent belongs to the same family; no reported result was found on Lacuna for it on these benchmarks.
For a fair comparison, both systems use the same backbone, the same stored traces, and the same $k$ exemplars in the final prompt. The pooled agent also receives a matched budget of model calls. Our bidding costs $N$ retrievals plus one generation, and the pooled agent spends the extra budget on additional retrieval rounds and on self-consistency voting over $N$ samples. Retriever and reranker hyperparameters are tuned on a shared validation split with the same number of trials for each system.
The strongest published structured-image system is PixelCraft, at 68.1% on CharXiv and 34.38% on the Geometry3K auxiliary-line subset with GPT-4.1-mini. We will rerun its released pipeline on our backbone where that is feasible. Otherwise we cite its numbers only as context, since its backbone differs from ours.
The key ablation is a topic-clustered split with the same $N$. Memory is assigned by k-means on the trace embeddings or by subject labels, and a nearest-centroid router sends each query to a cluster. This isolates whether competence-driven emergence matters, or whether any split of the memory would do.
We would count a clear win as follows. The split system beats the tuned pooled agent at the largest memory size by a margin exceeding the seed-to-seed spread across five stream orders, on MATH-Vision and on at least one chart benchmark. It also beats the clustered split at that size.
Why not just…?
Why not just use a bigger frontier model? Scale has not closed this gap. GPT-4o reaches 30.39% on MATH-Vision and GPT-4V reaches 48.1% on GAOKAO-MM, while FiD and IRCoT show that compact models with memory can beat models 14 to 58 times larger on text tasks. We report frontier numbers as a ceiling reference. The question we test is whether memory can move a small model toward that ceiling.
Why not just give one agent the whole memory with a good reranker? This is the real threat, and the pooled baseline is built to be as strong as we can make it. If $\Delta(S)$ stays at or below zero at all memory sizes, the proposal fails. In that case we report a negative result: multimodal procedural memory scales fine without specialization. The crossover prediction makes the outcome readable either way. Gains that appear only after pooled top-$k$ precision drops would support the limited-information explanation. Gains at every size would point to something else, such as an ensembling effect, which the matched self-consistency budget is designed to absorb.
Why not just split the memory by topic and route? If the clustered split matches the emergent one, emergence adds nothing beyond sharding, and we will say so. Our expectation, which the mutual-information analysis tests, is that competence-defined shards separate problems that look alike but are solved differently, for example two bar charts where one needs reading a value and the other needs computing a ratio. Clustering by appearance merges such problems.
Why not just hand-design perceiver and reasoner roles, as PixelCraft and MATA do? Hand-designed roles are orthogonal to our mechanism and could be stacked on top of it. They do not answer whether useful divisions of labor can be discovered from a stream of experience. That matters most where designers lack good intuitions, such as GeoSQA-style problems in which the bottleneck is applying domain principles.
Decisive Checks
Four results determine the outcome. The first is the shape of $\Delta(S)$ across the four memory sizes on MATH-Vision and CharXiv, averaged over five stream orders. The second is the emergent-versus-clustered comparison at the largest size. The third is the neutral-title Chart101 test, where memory that grounds answers in symbolic extraction should lose less accuracy than the backbone when titles are removed.
The fourth is a fragility test motivated by Miconi, who showed that specialist roles become single points of failure. We delete the largest agent's memory and measure how much accuracy drops and how quickly the remaining agents recover through continued streaming. A large, unrecoverable drop would argue for adding redundancy to the writing rule, so that verified traces are also copied to the runner-up bidder, and that variant would be tested before any broader claim.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Generalization through Memorization: Nearest Neighbor Language Models
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- When Is Diversity Rewarded in Cooperative Multi-Agent Learning?
- MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning
- PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images
- DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- Natural Emergence of Heterogeneous Strategies in Artificially Intelligent Competitive Teams
- Market Incentivization for Theorem Proving with LLM Agents
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic–Procedural Memory
- When Evolving Populations is Better than Coevolving Individuals: The Blind Mice Problem