Vision-language models, meaning language models that also read images by cutting them into small patches and turning each patch into a token, are now used to answer questions about long financial reports. On these tasks they hit a wall. On the cross-page memory level of the FinMTM benchmark, which requires linking evidence across pages of a long report, every one of 22 tested models scored below 50% (FinMTM). This note proposes that a large part of that wall is not a reasoning failure. It is a placement failure. When a report is fed in as page images, visual tokens make up almost the entire context, 97–99% by the estimate cited in CARES. The page holding the answer usually ends up deep in the middle of that sequence, and that is exactly where language models are known to recall poorly, a pattern called "lost in the middle."
If this diagnosis holds, two changes should help together. The first is splitting the job across agents: small "reader" agents each look at one page and write down short, structured evidence records with page and cell coordinates. The second is giving a separate reasoning agent only those records, placed where the model attends best. We test this on financial reports for two reasons: the evidence is dense in numbers and fragile to compression, and documents from different periods contradict each other as markets change. That second property turns stale-but-plausible evidence into a natural distractor.
Core Claim
We make three claims, each testable. First, on multi-page financial document question answering, a model's accuracy on a fixed question will depend strongly on where the evidence page sits in the visual sequence. The dependence will follow the U-shaped curve, high at the start and end and low in the middle, that Serial Position Effects of Large Language Models documented for text. Second, much of the gain that visual token pruning brings to long multi-image inputs comes from shortening the distance to the evidence rather than from removing noise. AVTP hints at this when it reports that pruning sometimes beats the unpruned model by bringing context back into the model's "comfort zone." Third, a page-reader / reasoner decomposition that passes provenance-tagged extractive records will beat both pruning and text-only retrieval. The margin should be largest on questions that need chart reading or cross-page binding, and on questions where an outdated period's value competes with the current one.
The closest systems either fix the visual token cost or fix position, never both. They also never ask whether one causes the other. Multimodal trading agents such as MM-DREX and FinAgent pass full chart images into the model, uncompressed. Pruning methods such as AVTP, DART and LOOK-M were measured on general images. Position fixes such as Positional Contrastive Decoding and MEAP were tested only on text.
Research Program
The first component is a position-controlled diagnostic. We render DocFinQA's full SEC filings (DocFinQA, 7,437 questions over filings averaging 123,000 words) as page images. Then, holding the question and evidence page fixed, we permute page order so the gold page appears at each decile of the visual sequence. We also run the same probe on FinMTM's 400 multi-page PDF reports, using its L4 memory questions. For each depth we record accuracy at full visual tokens and at AVTP-style 50% retention. The claim that pruning helps mainly through distance predicts a specific result. Pruning's benefit should be concentrated at middle depths and close to zero when the evidence page is already last. We will also check whether placement is the active ingredient by comparing pruning against a control that keeps the original sequence length but moves the gold page to the end.
The second component is the decomposed system. A page-reader agent sees one page at a time and is instructed not to reason, only to transcribe. This deliberately avoids the effect FinMTM reports, where long chain-of-thought degrades low-level visual perception. The reader emits records of the form (quantity, value, unit, period, page, row/column or chart-axis location). These are extractive, not summaries, because Characterizing Prompt Compression Methods found that extractive chunk reranking beats both abstractive summarization and token pruning. That paper also found token pruning breaks structured text, which the authors call the "grammar gap."
A cross-encoder reranker scores records against the question. The reasoner then receives the top records in an order that places the highest-scoring ones at the end of its prompt, nearest the point where it starts generating. Each record carries its period tag. Where several periods are present, the most recent is placed last and older values are labeled as superseded. This is our handle on market non-stationarity. In real filings, the prior year's figure for the same line item is the hardest distractor, and Beyond Continuity shows that models struggle to drop stale context.
Early perception errors otherwise propagate unchecked, a failure FinMTM traces to "initial misperception." To limit this, a verifier re-reads only the image crops cited by the records that the final answer actually uses, at high resolution. This borrows the coarse-then-zoom idea of VisionThink, but applies it after reasoning and only to load-bearing evidence. That keeps the extra cost proportional to answer complexity rather than document length. We count every token, including messages between agents, because AgentTaxo shows that inter-agent communication can erase efficiency gains.
All components run on open-weight backbones, Qwen3-VL-8B and InternVL3.5-8B, the two models AVTP reports on. This makes the comparisons directly anchored to that paper's settings.
Baselines
The naive reference is the monolithic model given all page images at full resolution in original document order. Its accuracy at each evidence depth defines the size of the problem.
The strongest existing efficiency method is AVTP. It is the strongest competitor because it is training-free, compatible with FlashAttention, and allocates its token budget per image according to relevance. It is therefore the method most likely to recover cross-page accuracy without any decomposition. AVTP reports keeping 96.1% of Qwen3-VL-8B's original accuracy and 94.1% of InternVL3.5-8B's at 50% visual token retention, with a 2x speedup, on general multi-image benchmarks. We found no reported result on Lacuna for AVTP on FinMTM or on rendered DocFinQA. We will therefore run it ourselves at 25%, 50% and 75% retention, with the same backbones and the same question sets. Beating it means higher accuracy at equal or lower total tokens, inter-agent messages included.
The strongest text-side competitor is OCR followed by the extractive 128-token chunk reranker of Jha et al.. On 2WikiMultihopQA they report 7.75x compression while increasing accuracy by 7.89 points. It gets the same reranker model and the same token budget as our records.
The position-only competitor is Positional Contrastive Decoding, applied to the monolithic model. It reports raising Llama-3-8B-262k on InfiniteBench from 72.0% to 79.0%. Its β parameter is tuned on the same development split, with the same tuning budget, as our reranker threshold.
Two ablations isolate the key ingredients. The first keeps decomposition but swaps extractive records for free-form page summaries, which tests whether the record format matters. The second keeps records but places them in random order, which tests whether placement matters. A third ablation lets the reader use chain-of-thought, which tests the FinMTM perception-pollution effect directly.
Why not just…?
Why not just prune visual tokens with AVTP and keep one model? This is the main threat. If our position-controlled diagnostic shows that pruning flattens the U-shaped curve as much as decomposition does, at similar cost, the multi-agent part is not justified and we will say so. The prediction that separates the two concerns placement. Pruning shortens the sequence but leaves the gold page in the middle, so accuracy at middle depth should still lag accuracy at end depth. Records placed last should remove that gap. AVTP's own evidence comes from general images. It does not tell us whether removing half the patches preserves thin chart features or single digits in tables.
Why not just run OCR and standard retrieval over text? For text-heavy filings this may tie our method. That would be informative: it would mean the visual pathway only matters for charts. We therefore report results split by question type: table-only, chart-dependent, and cross-page. A Table Is Worth 64 Tokens suggests that image form can be the more compact representation for tables. F²Agent found that serializing numbers as text lets textual bias drown quantitative signal. Neither result has been tested in this setting, so the split decides it.
Why not just reorder pages so the likely-relevant ones come last? This is cheap and tests our own hypothesis, so it is included as an explicit condition. If page reordering alone matches records-plus-placement, then position is the whole story and decomposition adds nothing. If records still win, the remaining gain comes from separating perception from reasoning and from the much shorter context.
Why not just use positional contrastive decoding? PCD's diagnosis is that the gold token is still among the top 8 candidates but is suppressed by nearby distractors. That may not hold when 99% of the context is image patches and the error is a misread number rather than a suppressed one. Running PCD on the monolithic model, with and without pruning, measures this directly.
Proposed Experiments
The decisive check is the depth curve. For each system we plot accuracy against the decile position of the evidence page on rendered DocFinQA and on FinMTM L4. The hypothesis survives only if three things hold. The baseline curve must be clearly U-shaped. Pruning must lift mainly the middle deciles. Records-plus-placement must flatten the curve at fewer total tokens than AVTP at 50% retention.
The second check is the drift probe. Where an earlier year's filing for the same company is available, we append it, so that a superseded value for the same line item is present. We then measure how often each system answers with the stale value. Period tags and recency placement should cut this error relative to every baseline. Uniform pruning, which cannot tell periods apart, should not.
The third check is the token budget, accounted end to end. If reader-agent messages push total tokens above those of the pruned monolithic model with no accuracy gain, the decomposition fails on its own terms.
A positive outcome on all three would show that, in this setting, visual token explosion and lost-in-the-middle are the same problem. A negative outcome on the first check would show the FinMTM memory wall is a reasoning limitation rather than a positional one, and that is worth knowing too.
References
- FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
- CARES: Context-Aware Resolution Selector for VLMs
- Serial Position Effects of Large Language Models
- Multi-Image Visual Token Pruning in Large Visual Language Models
- MM-DREX: Multimodal-Driven Dynamic Routing of LLM Experts for Financial Trading
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
- Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More
- LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
- Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
- DocFinQA: A Long-Context Financial Reasoning Dataset
- Characterizing Prompt Compression Methods for Long Context Inference
- Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
- VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
- AgentTaxo: Dissecting and Benchmarking Token Distribution of LLM Multi-Agent Systems
- A Table Is Worth 64 Tokens: Pixel-level Compression for Multi-Table Document Question Answering
- F$^2$Agent: Financial Fusion of Agentic Intelligence for Multimodal Trading