Scale-Step Experts: Distilling Slow Zoom Loops into Two-Pass Figure Reading

When the answer to a scientific question is not in the abstract, it is usually inside a figure or table. Abstract-only systems score below 10% on AirQA (AirQA), and roughly half of verified protein interactions appear only in full text (PreBIND and Textomy). The best current answer to this problem is an agent that repeatedly crops and re-reads sub-panels with a vision-language model (VLM), meaning a model that reads images and text together. It works: removing that zoom loop costs 13.5 points on figure questions. But it takes a median of six minutes per figure query (A Multimodal Literature Agent). This note proposes to compress that loop into two forward passes of a single VLM. The first pass, at low resolution, finds which panel or table cell holds the answer. The second pass reads the exact value from a high-resolution crop. Each pass uses its own small set of adapter weights.

The design borrows one lesson from diffusion models. The early, noisy stages of generation, which settle global layout, and the late stages, which settle fine texture, fight over shared weights. Giving each stage its own parameters helps, an observation made in ERNIE-ViLG 2.0 and formalized as gradient conflict in Decouple-Then-Merge. Our bet is that "where on the page is the evidence" and "what does this tick label say" conflict in the same way inside a document VLM. In our setting, zoom depth plays the role that diffusion time plays there. We do not use diffusion for extraction. We transfer the specialization principle and test whether it holds outside generation.

Core Claim

Train a VLM on question–figure pairs where both the evidence region and the answer are supervised. We claim that giving the localization stage and the reading stage separate low-rank adapters ("scale-step experts") yields higher figure and table QA accuracy than one shared adapter trained on identical data at identical compute. We further claim that the resulting two-pass system recovers most of the accuracy of a multi-round zoom agent using the same base model, while making far fewer VLM calls.

The first claim is the scientific one, and it can fail cleanly. If a shared adapter matches the split adapters, then the coarse/fine conflict seen in diffusion does not transfer to extraction. In that case, any speedup comes from distillation alone.

Research Program

The base model is Qwen2-VL-7B. It is the backbone used for document extraction in RIDGE and for timestep experts in LaTtE-Flow. The frozen-backbone design of LaTtE-Flow left understanding benchmarks unchanged, which suggests adapters can be added without damaging general skills.

The coarse expert sees the question and a downsampled page or figure. It emits a bounding box, either a region within one figure or a table row/column span, together with an answerability score. The fine expert sees the question and the high-resolution crop, and it emits the answer. Both experts are LoRA adapters attached to the same frozen backbone and switched on by stage, following the per-interval adapters of TimeStep Master. LaTtE-Flow flags an open problem: compute is split uniformly across stages for no principled reason. We address it directly by sweeping adapter rank independently per stage under a fixed total parameter budget. This tells us whether localization or reading deserves more capacity.

Supervision comes from distillation. We run the full zoom-loop agent, using the same base model and a frontier VLM as a teacher, over figure-grounded questions from VeriSciQA (dataset). VeriSciQA ties claims from figure-citing paragraphs to verified QA pairs. From each run we log the crop that finally yielded the correct answer, and that crop becomes the coarse-stage target. This mirrors AirQA's finding that fine-tuning on synthetic agent trajectories lifts smaller models. The difference is that we distill trajectories into a fixed two-stage architecture, not into a free-form agent.

The cross-modal gap enters as a design constraint, not as a separate module. A Visual Attention Grounding Neural Model showed that a single global image vector lets the decoder ignore the image, and that token-to-region grounding is necessary. Our coarse stage enforces that grounding explicitly: the question must be tied to a region before any reading happens.

Baselines

The naive reference point is the base VLM given the whole figure at its native resolution in one pass, with no adapters. Next to it we report the title-plus-abstract reference on AirQA, where AirQA reports below 10%. Together these mark the floor, and the single-pass VLM shows how much legibility alone limits accuracy.

The strongest existing method is the multimodal literature agent with its iterative VLM zoom loop and answerability reranking. On LAB-Bench 2 that paper reports 62.5% accuracy on FigQA2 and 88.8% on TableQA2 (A Multimodal Literature Agent). Those numbers come from a frontier model, so we will not compare a 7B system against them directly. Instead we re-implement the zoom loop on Qwen2-VL-7B with the same retrieval front end, and allow it up to its natural number of rounds. This matched-backbone zoom loop is the practical target. Beating it means coming within 3 points of its FigQA2 accuracy while using at least five times fewer VLM calls and less wall-clock time. Matching or exceeding its accuracy would be a strong result. We also report the published frontier figure as context.

The ablation that isolates the key ingredient is a shared-adapter model. It uses the same two-pass inference and the same distilled data, the same total LoRA parameters, and the same training steps and learning-rate search. The only difference is that one adapter serves both stages. A second ablation removes the coarse stage and fine-tunes a single adapter end-to-end on full-figure QA, which isolates the value of explicit localization. For AirQA we additionally cite the reported 44.14% for the top GPT-4o agentic setup and 35.07% versus 20.22% for agentic hybrid versus standard RAG on Qwen2.5-72B. These place our numbers on the existing scale. All tuned variants receive the same hyperparameter search budget.

Why not just…?

Why not just run a detect-then-crop pipeline with an off-the-shelf layout detector and a generic VLM? This is the most serious threat, and the proposal is built to measure it. Layout detectors, including the spatial-preserving parsing in Detect, Retrieve, Comprehend, find every figure or table. They do not find which sub-panel answers this question. The literature agent's own ablation shows that answerability, not similarity, drives table accuracy: removing its reranker costs 8.3 points. We will run a detector-plus-crop baseline that enumerates all detected panels and lets the untuned VLM pick, at matched call budget. If it matches the learned coarse expert, the localization stage adds nothing and we will say so.

Why not just use one fine-tuned adapter for both stages? This is the shared-adapter ablation above. It is the direct test of whether temporal specialization transfers. We will also measure the cosine similarity between localization-loss and reading-loss gradients inside the shared adapter, following the gradient-conflict diagnosis of Decouple-Then-Merge. A consistent negative or near-zero cosine together with an accuracy gap supports the claim. Positive alignment with no gap refutes it.

Why not just feed full pages to a large long-context VLM at high resolution? The evidence says scale alone does not close the gap. SciAssess finds frontier models degrade sharply on multimodal reasoning tasks. GPT-4o tops out at 44.14% on AirQA. The literature agent reports that single-pass ingestion leaves sub-panel labels illegible. Our single-pass baseline at maximum native resolution tests this on our own data. If it matches the two-pass system, the contribution reduces to cost and not accuracy.

Why not just convert figures and tables to text with OCR and parsing heuristics? Flattening tables destroys row–column structure (TabPrompt). OCR noise collapses discriminative extractors: LayoutLMv2 loses about 40% F1 at 50% character noise in GenKIE. Gels and micrographs have no faithful text form. We include an OCR-to-markdown baseline on TableQA2, where it is most competitive, and expect it to trail on FigQA2.

Proposed Experiments

Training uses VeriSciQA trajectories. Evaluation uses LAB-Bench 2 FigQA2 and TableQA2, plus the table/figure/formula subset of AirQA, since all three consist of questions whose answers lie outside abstracts. We report accuracy, VLM calls per question, and wall-clock latency. We also report intersection-over-union between the predicted coarse region and the teacher's final crop, which shows whether failures come from localization or reading.

The first decisive check is the split-versus-shared comparison at matched parameters and data, with three seeds. We will only claim a real gap if it exceeds seed variance on both FigQA2 and AirQA.

The second check is the capacity sweep. If accuracy is maximized by giving most rank to the coarse stage, then localization is the bottleneck and future work should target it. A tilt toward the fine stage would point to reading.

The third check is the accuracy–cost curve against the matched-backbone zoom loop. We plot the zoom loop at one, two, and full rounds against our two-pass system. This shows whether two specialized passes dominate an equally cheap truncated loop. If they do not, the method has no practical case regardless of the ablation result.