Strip-Shaped Evidence Windows for Multi-Hop Document Question Answering

Questions about financial reports, scientific papers and chart-heavy documents often need several reasoning steps. The system reads one table row, uses that value to find a chart series, then looks up a column header somewhere else. Current systems fail mostly at the looking-up step: they grab the wrong cell, bar or line. Most of the evidence they need is long and thin, like a table row, a column, a legend entry or a bar. The vision models that read these pages, though, cut every page into small square tiles and attend to square neighborhoods around them. We propose that each reasoning step should return a rectangle of the page whose height and width are chosen for that step: a wide, short strip when the step asks for a row, a tall, narrow one when it asks for a column. Only that strip goes to the answering model. We also propose to test directly whether this shape-matching, and not merely zooming in, is what fixes the errors.

The opportunity matters because the field holds a puzzling contradiction. Retrieving whole pages as images beats text extraction by a wide margin. Yet feeding page images into multi-step reasoning often makes answers worse than feeding plain text. If part of that penalty comes from square visual regions mixing neighboring rows and series together, the fix is cheap and lives in the retrieval loop, not in retraining a vision backbone.

Core Claim

Our hypothesis is that the "alignment tax" reported by DocHop-QA is partly a geometric crosstalk problem. That paper found that adding page images to vision-language models often degrades multi-hop answers relative to text, and that evidence bounding-box localization is very poor. By crosstalk we mean that square regions large enough to contain a full row or bar also contain adjacent rows or series. The same goes for the chart attribution errors in DocHop, where accuracy falls from about 85% on 2-chart pages to about 40% on 6-chart pages as models pick the wrong bar or line.

The proposal is an iterative retrieve-and-reason loop in the style of IRCoT. IRCoT alternates generating a reasoning sentence with using it as a search query. In our version, each intermediate reasoning step is scored against the per-tile embeddings of a visual page retriever, and the loop then selects an axis-aligned rectangle with a free aspect ratio. That rectangle is re-encoded at high resolution and becomes the only new visual context for the next step.

What is new is narrow and testable. The shape of the region passed forward is set by the current reasoning step. We isolate that shape against square regions of equal area and equal token cost.

Where the Pieces Come From

Two strands of computer vision supply the mechanism.

The first is about rectangular receptive fields. Deep Visual Template-Free Form Parsing replaced square 3×3 kernels with dilated 1×3 kernels so that long text lines were not split into fragments. Detection on historical forms rose from 0.324 to 0.421 mAP, and F-measure from 0.612 to 0.808, with no added parameters. That result motivates us: it is direct evidence that aspect-ratio mismatch hurts document evidence localization. Adaptive Rectangular Convolution learns only a height and a width per location instead of free offsets for every sampling point, and reports better convergence than unconstrained deformable convolution. We reuse its idea that two geometric numbers are enough and are easier to learn than arbitrary deformations.

The second strand shows that crosstalk is real for elongated targets. Dynamic Refinement Network shows that square receptive fields stretched to cover elongated objects absorb clutter. Oriented-convolution work such as RMSIN rotates filters toward runways and ships, but only for single-step referring segmentation. Our setting differs from all of these in one respect: the geometry is chosen per reasoning step from a query, not per pixel from image content alone.

On the reasoning side, CoRAG trains sub-query, sub-answer and final-answer stages and raises MuSiQue retrieval recall from 29.0% to 47.1%. It is text-only. ColPali indexes pages as images with multi-vector late interaction, meaning each query word is matched against every image tile and the best matches are summed. It reaches 81.3 NDCG@5 on ViDoRe against 66.1 for an OCR plus text-retrieval pipeline, but it retrieves whole pages in a single shot. We keep ColPali's tile-level similarity maps. We stop treating the page as the unit of retrieval and treat a step-specific rectangle as the unit.

Research Program

The work has three stages. The first two are cheap and decide whether the third is worth doing.

Stage one is a crosstalk diagnostic. On DocHop chart pages and on rendered tables, we perturb the content adjacent to a target element, such as the neighboring row's numbers or the next bar's height. We then measure how much the representation of the target changes in three cases: square-tile embeddings from the reader model, square crops, and strip crops. If square representations of the target barely move when neighbors change, crosstalk is not the problem and the proposal should stop or be reframed.

Stage two is a training-free strip selector. For a sub-query $q_t$ at hop $t$, ColPali gives a similarity $s_t(p)$ for every page tile $p$, taking the maximum over query words. We choose the rectangle

$$R_t^* = \arg\max_{R} \; \frac{1}{|R|}\sum_{p \in R} s_t(p) \; - \; \lambda \log |R|,$$

where $R$ ranges over axis-aligned rectangles with any integer height and width, $|R|$ is the number of tiles in $R$, and $\lambda$ trades coverage against compactness. The search is exhaustive and cheap on a tile grid. The selected rectangle is re-cropped from the original page at a fixed visual-token budget and passed to the reader along with the running reasoning trace. The reader is Qwen2.5-VL, as used in DocHop-QA. The loop terminates as in IRCoT.

Stage three is a learned shape head, run only if stage two shows a shape effect. In the spirit of Adaptive Rectangular Convolution, a small head predicts height and width from the sub-query and the similarity map. It is trained on DocHop-QA's evidence boxes and on box labels derived from table structure. Because it only predicts a center plus two side lengths, it avoids the instability of free-form offsets.

We deliberately do not replace the square-tile tokenizer of the backbone. That is much more expensive, and it is only justified if the cheaper intervention shows shape matters.

Baselines

The naive reference point is single-shot ColPali page retrieval with the whole retrieved page passed to the same Qwen2.5-VL reader. This is the "do nothing" pipeline. DocHop-QA reports Qwen2.5-VL generative answering at BLEU 21.56 and ROUGE-L 23.90, and LayoutLMv3 evidence localization at 12.35 F1. These are the reference numbers for that benchmark. Beating them on localization F1 is necessary but not sufficient.

The strongest existing method for the multi-hop part is IRCoT run over OCR-serialized text, with tables serialized as Markdown and the same reader in text mode. It carries the full force of iterative retrieval and avoids the image penalty entirely. IRCoT's own text-only gains of up to 15 F1 points and its 3B-beats-175B result show why it is the competitor to fear. We run it with the same hop limit, the same total context budget and the same prompt-tuning budget.

A second strong competitor is visual IRCoT over whole pages. Each hop retrieves a page with ColPali and appends it. This isolates the value of iteration with images but without regions.

For external context on single-page charts, DocHop reports GPT-5.2 Reasoning at 60.18% and Gemini-2.5-Pro-Reasoning at 44.24%, with open-source models between 7% and 22%. Our open-source pipeline counts as clearly successful if it closes a substantial part of the gap to the proprietary models, especially on the 6-chart split.

The key ablation is the same loop with the rectangle forced to be square, with area equal to the selected strip and the same re-encoding token budget. This separates "shape matters" from "zooming in matters." A second ablation lets the region be unconstrained: the top-$k$ tiles scattered anywhere, mimicking deformable sampling. This tests whether the rectangular constraint helps.

All variants share retriever weights, reader weights, hop budget, visual token budget per hop and the number of tuning runs for $\lambda$. Beating the square-crop ablation by a margin larger than its seed-to-seed variance, concentrated on high-aspect evidence, is the result that counts.

Why not just…?

Why not just use OCR text with IRCoT or CoRAG? This is the cheapest strong alternative, and on table-only questions it may well win. ChartQA reports that models working from extracted tables fail on questions that refer to visual properties such as colors or the tallest bar. ColPali reports that OCR pipelines are over 18 times slower to index and retrieve worse. Its 7.2 seconds versus 0.39 seconds per page is an indexing cost, though, not an answer-quality argument. We report results split by evidence type, so if text IRCoT matches us on tables we will say so and limit our claim to charts and mixed layouts.

Why not just crop and zoom with square regions? This is the real threat, because much of any gain could come from higher effective resolution rather than shape. The equal-area, equal-token square ablation exists for this reason. Our prediction is that strips beat squares mainly when the gold evidence box has an aspect ratio far from one, and on denser pages. If the advantage is uniform across aspect ratios or absent, the shape hypothesis fails and the contribution reduces to "iterative zooming," which we would report as such.

Why not just fine-tune the vision-language model or rely on long-context models? Self-attention over square tiles can in principle learn row-shaped dependencies. Long-context frontier models see the whole page. DocHop shows frontier reasoning models at 60.18% and below on single-page multi-chart documents, against 92.60% for humans, so whole-page attention alone does not solve attribution. A fine-tuned reader is still a fair competitor. We therefore include a variant that fine-tunes the whole-page reader on the same supervision the shape head sees, to check that any gain is not just extra training signal.

Why not just use deformable convolutions or free-form region selection? The scattered top-$k$ ablation covers this. Adaptive Rectangular Convolution reports that constraining geometry to height and width beats free offsets for stability. We test whether that holds when the geometry comes from a query rather than from image features.

Proposed Experiments

The main benchmarks are DocHop and DocHop-QA. DocHop has 1,876 single-page, multi-chart examples, and we report accuracy split by chart count. DocHop-QA has 11,379 multi-document instances, and we report evidence-box F1, BLEU and ROUGE-L, separately for its "fan-hop" questions (aggregating several facts) and "chain-hop" questions (each step depends on the previous one).

For tables, we use TAT-QA and FinQA with tables rendered as page images. These numbers will not be directly comparable to the text-input results reported there: TAGOP at 58.0 F1 and FinQANet at 61.24% execution accuracy. Their error analyses, which attribute 55% and 15% of errors respectively to evidence extraction, define the subset we target. We also run ViDoRe page retrieval to confirm that strip selection leaves first-hop retrieval near ColPali's 81.3 NDCG@5.

Three checks are decisive. First, the strip-over-square gain should grow with the aspect ratio of the gold evidence box, and it should grow from 2-chart to 6-chart DocHop pages. Second, in stage one, perturbing neighboring rows or series should move square representations of a target more than strip representations. Third, the loop should cut accumulated visual tokens per question relative to whole-page visual IRCoT while keeping or improving accuracy. That addresses the context-growth problem IRCoT itself flags.

A clean negative result on the first two checks would also be informative. It would tell the document-understanding community that the alignment tax is not a receptive-field problem, and that effort should go elsewhere.