Many questions about photographs of the real world need several readings in sequence. To answer "which floor do I go to for my flight?", you read the gate on the departure board, then find that gate on a directory sign, then read the floor next to it. Each reading tells you what to look for next. Current systems handle this poorly. A text spotter (a model that finds and reads every piece of text in an image) runs once over the whole picture. Every string it produces, relevant or not, then goes to a reasoning model. That design pays twice. Any misread word that a later step depends on breaks the chain. Every irrelevant word adds a distractor, and distractors are known to hurt more as the chain gets longer.
We propose that the reasoning model should drive the reading. After each reasoning step, the system uses that step's sentence to find the one or two places in the image that matter next. It crops and straightens those text regions and re-reads them at higher resolution. Only those strings enter the context. This is the image analogue of interleaved retrieval for multi-hop question answering, with the image as the corpus and the spotter as the retriever. The main claim is measurable: read-on-demand should flatten the rate at which accuracy decays with the number of hops. A controlled benchmark should also let us split the remaining decay into a part caused by misreading and a part caused by reasoning.
Core Claim and Where It Comes From
Three bodies of evidence motivate this. The first is from text-only question answering. IRCoT generates a chain of thought one sentence at a time and uses each sentence as the next retrieval query. It reports retrieval recall gains of 11–21 points and question-answering F1 gains of up to 15 points on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. The reason is that the entity needed at hop two is unknown until hop one is solved. IRCoT only retrieves passages from a text collection, and we reuse its loop structure over pixels.
The second is the work on reasoning decay, which shows that long chains fail even when each step is easy. That work spans fictional-ontology proofs in PRONTOQA-OOD, graph-structured tasks in DyVal, and synthetic visual scenes in the LoRA visual question answering benchmark, where zero-shot InstructBLIP falls to 30.5% on queries needing more than six inference steps. GSM-DC adds the key fact for our setting. Error grows as a power law in the number of irrelevant context items, and the damage grows with depth: on 5-step problems, Grok-3-Beta's step accuracy falls from 43% with 1 distractor to 19% with 15. A busy street scene can easily contain dozens of irrelevant strings.
The third is scene-text question answering, which has never tested depth. LoRRA/TextVQA, ST-VQA, LaTr, LOGOS and LTG all consume a single, fixed dump of text from one OCR pass. LTG comes closest to our idea: it first locates one answer box and then generates the answer. It assumes a single target region and cannot use a first reading to find a second. LaTr's authors note that current benchmarks test single-step reading.
A simple model makes the claim precise. Let $A(d)$ be the accuracy on a question that needs $d$ hops. Let $r_h$ be the probability that the text needed at hop $h$ is read correctly, and let $s_h(n)$ be the probability that the reasoning step at hop $h$ is correct when $n$ irrelevant strings are in context. If the hops fail roughly independently, then
$$A(d) \approx \prod_{h=1}^{d} r_h \, s_h(n).$$
Under this model, $\log A(d)$ is roughly linear in depth, and the slope is set by the per-hop terms. A one-shot pipeline fixes $r_h$ at whatever the spotter achieves at base resolution and sets $n$ to the whole scene. Read-on-demand raises $r_h$ by re-reading and shrinks $n$ to a handful of strings. We predict a flatter slope. We do not assume independence: we test it by comparing the measured $A(d)$ against the product of separately measured per-hop rates.
Research Program
Benchmark. The first artifact is a benchmark with controlled depth, which we call ChainText. Each question is a pointer chase through text in an image: a key read in one region names the region holding the next key, and so on for $d$ hops, with $d$ from 1 to 6. Scenes are composed from real text crops taken from Total-Text, CTW1500, Inverse-Text and ICDAR 2015. The crops are placed onto rendered signs, departure boards, menus and directories. Using real crops keeps reading genuinely hard, because curved, inverted and incidental text all appear. Three factors are varied independently: depth $d$, the number of distractor strings $n$, and reading difficulty (horizontal versus curved or inverted crops). Because every chain is generated, we know the ground-truth text and location for every hop. This makes it possible to replace any single component with an oracle.
To check that conclusions carry over from synthetic scenes, we also hand-write a smaller set of 2–4 hop questions on real TextVQA and ST-VQA images that contain several text instances.
Method. The read-on-demand system has three parts.
- Coarse pass. An arbitrary-shape spotter runs once at base resolution and produces a coarse map of the image: each text instance's boundary plus a tentative transcription. We use DNTextSpotter, which describes each text instance by Bézier control points along its centre line.
- Locating the next region. At each hop the reasoning model writes one sentence naming what it needs, for example "find the row labelled B12". That string is matched against the coarse transcriptions by normalized edit distance. Edit distance tolerates the misreads the coarse pass will make; the soft-matching idea follows ST-VQA's ANLS metric. Layout cues such as "same row" or "the arrow points left" can widen the set of candidate regions.
- Re-reading. Each selected region is cropped along its curved boundary, straightened into a flat strip, upsampled two to four times, and re-read. For the re-reading we use TiCLS, whose character-level language model was pre-trained on sequences of up to three words. Those are exactly the short strings that crops contain. TiCLS adds 2.73× parameters and 1.55× inference time over DeepSolo, but here it runs only on a few crops.
The re-read strings and their immediate neighbours are the only text added to the context. If no candidate matches above a threshold, the system re-spots tiles of the image at higher resolution and searches again. This is a minimal form of backtracking, which existing interleaved systems lack.
Reasoner as a factor. The reasoner is a separate experimental factor. We run the loop with an instruction-tuned language model writing the chain of thought, and also with a symbolic engine in the style of VIEIRA. VIEIRA runs foundation models inside a probabilistic Datalog engine (a database-style logic language that chains facts) and reaches 100% on Tracking Shuffled Objects, against about 84% for GPT-4 with chain of thought. Mirzaee et al. report 96–97% on SPARTQA-Auto by separating extraction from symbolic reasoning. Those results constrain what we can claim. If a symbolic reasoner removes the reasoning term, then whatever decay remains is the reading term. Read-on-demand should still help there, because it acts on reading, not on reasoning.
Analysis. The analysis fits a slope of $\log A$ against $d$ for each system, condition and reading-difficulty level. It then decomposes the decay by substituting oracle text (setting $r_h = 1$) and oracle distractor removal (setting $n = 0$), one at a time and together.
Baselines
Naive reference. The do-nothing reference is the obvious pipeline. DNTextSpotter runs once at base resolution, and every recognized string, each with its box coordinates, goes into the same language model with chain-of-thought prompting. This baseline isolates our whole contribution, because the reasoner, the spotter weights and the prompt format are identical; only the timing and resolution of reading differ. For context on spotter quality, DNTextSpotter reports lexicon-free spotting accuracy of 84.5% on Total-Text and 67.0% on CTW1500, and 75.9% F1 on Inverse-Text, in its paper. Those rates on the curved and inverted subsets suggest that misreading will compound visibly by three or four hops.
Strongest learned single-pass model. This is LaTr, a layout-aware T5 that reads question, OCR tokens with their coordinates, and image patches in one pass. We fine-tune it on the ChainText training split, using the same number of tuning runs as for our system. The file reports only gains for LaTr: +7.6 absolute on TextVQA and +10.8 on ST-VQA. For an absolute reference on these benchmarks we use LOGOS, which reports 51.08% TextVQA test accuracy and 0.579 ANLS on ST-VQA. No reported result was found on Lacuna for any system on multi-hop scene-text questions, so every number on ChainText will be new.
End-to-end vision-language model. We include the strongest openly available instruction-tuned vision-language model, run zero-shot and also fine-tuned on ChainText. The reference point is InstructBLIP's 41.2% average and 30.5% beyond six steps on LoRA.
Symbolic one-shot pipeline. This is the most dangerous competitor on synthetic chains: one-shot spotting, followed by extraction of key–value relations, followed by Datalog in the VIEIRA style. It is run with exactly the reasoner used in our symbolic variant.
Ablations. Two ablations isolate the key ingredients. The first is hop-conditioned pruning without re-reading: coarse strings selected by the same matching, but with no crop and no upsampling. It separates the distractor benefit from the reading benefit. The second compares axis-aligned crops with Bézier-straightened crops during re-reading. It tests whether the curved-geometry representation matters once reading is targeted at a few regions.
Fairness. All systems share training data and tuning budget, and every comparison reports accuracy against total spotter compute in floating-point operations (FLOPs). Beating a baseline means a significantly smaller decay slope, measured over three or more seeds and with confidence intervals obtained by resampling questions. It also means equal or better accuracy at $d = 1$ at matched compute. A system that only helps at depth one is not testing our claim.
Why not just…?
Why not just give the full OCR dump to a strong language model with chain of thought? This is the naive baseline, and GSM-DC predicts how it fails: distractor damage grows with depth, so a long chain over a cluttered scene should decay steeply. The pruning-only ablation shows how much of our gain comes from removing distractors alone. The oracle-text condition shows how much comes from misreads the model cannot repair. If a frontier model's slope stays flat as $n$ grows, the distractor half of our argument fails, and we will report that.
Why not just run the spotter at high resolution over the whole image once? This is a real threat to the reading half of the claim. Full high-resolution spotting raises $r_h$ for every string, but it also raises compute for every string and leaves $n$ unchanged. We run it as an extra baseline and plot it on the same accuracy-versus-compute curve. If it matches read-on-demand at equal compute with a symbolic reasoner, our contribution reduces to an efficiency result, and we will say so.
Why not just use a symbolic reasoner and skip the loop? On clean inputs, the work of Mirzaee et al. and VIEIRA suggests a symbolic reasoner removes reasoning decay. But a symbolic chain is only as good as its extracted facts: one misread key breaks it. That is why the reasoner is a factor in our design rather than a rival. The decisive comparison is symbolic one-shot against symbolic read-on-demand on the curved and inverted subsets.
Why not just filter the OCR tokens by question relevance once, or cluster them as LOGOS does? A single relevance pass cannot know the hop-two key before hop one is read, which is exactly the multi-hop gap IRCoT addresses. LOGOS's clustering by spatial density groups nearby words, but our chains connect regions that are far apart. A one-shot relevance filter, keeping the top-$k$ strings by similarity to the question, is included as an extra ablation. We expect it to match our system at $d = 1$ and fall behind as $d$ grows.
Proposed Experiments and Decisive Checks
Main experiment. We train and evaluate all systems on ChainText with $d \in \{1,\dots,6\}$, $n \in \{0, 5, 15, 40\}$, and two reading-difficulty levels. We report exact-match and ANLS answer accuracy, per-hop accuracy, and the fitted decay slope.
Check 1: does the slope flatten? The first decisive check is whether read-on-demand's slope is flatter than the naive baseline's and the one-shot symbolic pipeline's, with the gap largest on curved and inverted text and at large $n$. A flatter slope only on easy horizontal text would point to distractor removal rather than re-reading.
Check 2: decomposition. We compare oracle-text and oracle-pruning conditions to apportion the decay between reading and reasoning. We then test whether the product of separately measured per-hop rates predicts the measured $A(d)$. If the hops are strongly correlated, errors are not independent. That would itself be a finding about how misreading interacts with reasoning drift.
Check 3: transfer to real images. On the hand-written real-image set, read-on-demand must keep its ordering against LaTr and the naive pipeline. We also report standard TextVQA and ST-VQA accuracy to confirm that single-hop reading has not regressed.
Interpreting a negative result. If high-resolution one-shot spotting with symbolic reasoning matches us everywhere at equal compute, the conclusion is that reading, not reasoning, drives multi-hop scene-text decay, and that targeted reading only saves compute. That is a useful and publishable answer to the open question of whether this decay comes mostly from misreading or mostly from reasoning.
References
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering
- How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
- Towards VQA Models That Can Read
- Scene Text Visual Question Answering
- LaTr: Layout-Aware Transformer for Scene-Text VQA
- Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
- Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA
- DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training
- TiCLS: Tightly Coupled Language Text Spotter
- Relational Programming with Foundational Models
- Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning