Carrying What the Model Saw: Verifiable Perceptual State for Multi-Turn Chart and Table Dialogue

When an analyst talks to a vision-language model about a financial chart over ten turns, the answers get worse as the conversation goes on. The cause is usually not a lack of reasoning ability. A number misread in the first turn gets silently reused, and the model's attention spreads thinner over a growing transcript. The FinMTM benchmark traces most multi-turn failures to exactly this "initial misperception" that cascades across later turns (FinMTM). ProBench documents a steep accuracy penalty when expert visual tasks are embedded in conversations of up to 13 turns (ProBench). We propose to train a compact (7–8B) vision-language model to keep an explicit observation ledger: a short, structured record of every value it has read off the image, including the number, its unit, and where on the chart or table it came from. The model carries this ledger forward instead of the raw history, and it can revise an entry by looking at the image again.

Because chart and table values have ground truth, every ledger entry can be checked automatically. That turns the sparse "was the final answer right?" signal of multi-turn reinforcement learning into a dense, per-turn reward on perception itself. The bet is simple. A conversation degrades because of what the model believes it saw, so reward the model for keeping that belief correct and editable, and the multi-turn penalty should shrink, especially in a small model specialized to one domain.

Core Claim

A compact model trained with multi-turn reinforcement learning, rewarded both for final answers and for the verified correctness of a persistent, editable perceptual record, will lose substantially less accuracy across turns than three alternatives. The first is the same model with a full transcript. The second is the same model prompted to keep such a record without training. The third is the same model trained with outcome-only multi-turn rewards. On first-turn-error recovery, the trained model should also outperform frontier long-context models. The new ingredient is not structured grounding, which already exists, and not multi-turn RL, which also exists. It is the use of a verifiable perceptual state that persists across turns as both the memory and the reward target.

The closest work splits along exactly this line. Think with Structured Grounding (TwSG) trains an 8B model to emit <observation> blocks with JSON coordinates and optimizes them with tag-level policy optimization. Its observations are discarded after a single question, and its reward is on the answer. MURPHY extends group-relative policy optimization (GRPO, a reinforcement learning method that scores each sampled response against others drawn for the same prompt) to multi-turn rollouts. It propagates the best downstream reward back to earlier turns, but it works only on text code with unit tests. Explicit Modular Decomposition (EMD) keeps dialogue cost constant through keep/update/delete edits on a text slot state, but its slots come from a fixed ontology with no visual grounding. ContextQFormer remembers past turns as pooled [CLS] embeddings, which throw away the coordinates and cell values that chart arithmetic needs. SegLLM feeds past masks and boxes back into the model, but only for segmentation and without numeric state. Our proposal borrows the observation format from TwSG, the edit operations from EMD, and the multi-turn credit assignment from MURPHY. It adds what none of them has: a per-turn reward computed from the ledger against source data.

Research Program

The ledger. At each turn the model emits three things before answering. It writes ledger operations (add, keep, revise, delete). Each entry names a quantity ("Q3 2023 revenue, segment B"), gives a value and unit, and gives a location: a bounding box for charts or a row/column reference for tables. It then writes brief reasoning and the answer. On the next turn, the image stays in context, but past turns are replaced by the ledger plus the last user message. A revise operation must come with a new <observation> of the cited region, so correction requires looking again, not just re-asserting.

Data with verifiable values. We build multi-turn conversations over images whose underlying data is known. The sources are ChartQA charts with their source tables (ChartQA), PlotQA and DVQA-style synthetic plots, and table images from TAT-QA, TABMWP and WikiTQ as used in TABQAWORLD. A conversation is a chain of single-turn questions linked by coreference ("and how does that compare to the year before?"). Later turns depend on values extracted earlier, and they include the multi-cell ratio and difference operations that TABQAWORLD reports remain a bottleneck. A fraction of conversations contain a user challenge ("are you sure that bar is 42?") or a deliberately ambiguous first-turn read, so that revision is sometimes needed. For the finance regime, we draw charts and report pages from the FinMTM training-style distribution where licensing allows, keeping FinMTM's evaluation split untouched.

Training. We use a cold-start supervised stage on teacher-written ledger conversations. GPT-4o-class teachers, as in TwSG, write the conversations, and entries are filtered against ground truth so that the teacher's own misreadings are removed. This is followed by multi-turn GRPO. The per-turn reward is

$$r_t = \mathbb{1}[\text{answer}_t \text{ correct}] + \lambda \cdot \frac{1}{|L_t|}\sum_{e \in L_t} \mathrm{match}(e, \mathrm{truth}),$$

where $L_t$ is the ledger after turn $t$, $\mathrm{match}$ scores an entry's value within a relative tolerance and its location by box overlap or exact cell match, and $\lambda$ weights perception against outcome. To pass credit backward across turns we use MURPHY's max-descendant rule. To keep long reasoning segments from swamping short ledger and answer segments in the gradient, we use TwSG's tag-level importance sampling, which exists for exactly that length bias. The entry-level reward is the piece we contribute.

Analysis. Beyond accuracy, we measure the multi-turn tax directly. Each final-turn question is also asked in isolation with the needed context written out in words, and the tax is the difference between the two accuracies. We plot it against turn index. We also log ledger accuracy per turn, which lets us separate "misread and never fixed" from "read correctly but reasoned wrongly." This mirrors the vision-versus-reasoning split of MATH-Vision, which attributed 31.9% of GPT-4V errors to recognition and 42.2% to reasoning, but here we track the split over time.

Baselines

The naive reference is the base compact model, Qwen2.5-VL-7B or the TwSG-8B checkpoint if available. It receives the full transcript and image at every turn, with no training beyond its release. It anchors the tax measurement.

The strongest existing compact method is TwSG-8B used turn by turn with full history. It reports 73.70% average accuracy across 11 single-turn visual-tabular subsets and 67.8% on CharXiv-R, against 56.7% for GPT-4.1 (TwSG). Because it already emits structured observations, it is the competitor most likely to make our ledger look unnecessary. For the strongest multi-turn training method, we re-implement MURPHY-style multi-turn GRPO with outcome-only rewards on our multimodal conversations. MURPHY reports up to 8% absolute pass@1 gain over compute-matched single-turn GRPO, but only on code (MURPHY); there is no reported multimodal result. We also include ContextQFormer, which reports a 68.17% available rate on TMDialog against 64.01% for LoRA-only fine-tuning (ContextQFormer). As a frontier reference, we run GPT-4o and Gemini with full transcripts. FinMTM reports that every tested model scored below 50% on its L4 memory level. ProBench's multi-round track has no reported per-model number on Lacuna (no reported result found on Lacuna for ProBench multi-round ELO values), so we rerun its judge protocol ourselves.

Fairness means four things. All trained variants start from the same checkpoint and the same conversations. They get the same number of GRPO rollouts and gradient steps. They get the same hyperparameter search budget, eight configurations each, selected on a shared validation split. They also run under the same context budget at inference. Three ablations isolate the key ingredient: ledger format with outcome-only reward ($\lambda=0$); ledger reward but no revise operation; and full transcript plus ledger reward, with no history replacement. Beating the field means two results. On FinMTM L3 (self-correction) and L4 (memory), the full system must outperform outcome-only multi-turn GRPO by a margin larger than the seed-to-seed spread over three seeds. It must also reduce the measured multi-turn tax at turns 6–10 by at least half relative to TwSG-with-history, while losing no more than a point of single-turn ChartQA and CharXiv-R accuracy.

Why not just…?

Why not just give a frontier model the whole conversation? Long context windows do not fix what is being remembered. FinMTM reports every tested model, frontier included, below 50% on cross-page memory tasks, and attributes failures to uncorrected first-turn misreads. ProBench reports the multi-round penalty across 24 models. We still run this baseline, because a frontier model may beat our compact model on absolute accuracy. Our claim concerns the slope of decay and recovery from injected first-turn errors, and if the frontier slope is as flat as ours, the claim fails.

Why not just prompt the model to keep a JSON summary each turn? This is the most serious threat, and it is why prompted-ledger appears as a baseline. The structure alone might carry most of the benefit. Two pieces of evidence argue against that. First, a compact model's summaries inherit its misreads, and nothing in prompting pushes it to re-inspect. Second, VisChainBench finds that small (~3B) models lean on language priors rather than the image in multi-turn chains (VisChainBench). If the prompted ledger on the base model matches the trained system on FinMTM L3/L4, the contribution reduces to a prompting recipe, and we will report it as such.

Why not just apply a single-turn grounded model like TwSG at every turn? TwSG grounds each answer but rebuilds its observations from scratch each turn while still reading the full, growing transcript. That leaves both attention dilution and cascading misreads untouched. The TwSG-with-history baseline and the full-transcript ablation measure this directly. If they match ours, persistence of state adds nothing beyond per-turn grounding.

Why not just do multi-turn RL on final answers, as MURPHY does for code? In code, a failing test tells the model what went wrong. In chart dialogue, a wrong final answer at turn 7 does not say whether turn 1's read or turn 7's arithmetic was at fault. The $\lambda=0$ ablation tests whether the dense perceptual reward is what makes credit assignment work.

Why not just convert charts to tables with OCR and use a text model? ChartQA shows that table-only models fail on questions that refer to visual features. TABQAWORLD reports a 5.42% gain from keeping visual representations over fixed-modality ones. GeoSQA shows that even perfect text descriptions do not close the reasoning gap (GeoSQA). Our ledger is itself a partial linearization, but it stays tied to image regions that the model can revisit.

Proposed Experiments and Decisive Checks

The main evaluation runs on four suites. The first is FinMTM, reported by level: L1 comprehension, L2 calculation, L3 self-correction and L4 memory. The second is the ProBench multi-round track, scored with its pairwise judge. The third is our held-out chained-ChartQA/TAT-QA conversations, with exact-match accuracy by turn index. The fourth is single-turn ChartQA, CharXiv-R and MM-Vet, used to measure any loss of general ability that specialization causes. The specialization tax is a known risk: Specializing Smaller Language Models reports collapse on BigBench Hard after math distillation.

Three checks are decisive. The first is the injected misread test: we overwrite one turn-1 ledger entry (or, for transcript baselines, the corresponding turn-1 statement) with a wrong value, then measure how often later turns still reach the correct answer and how often the model issues a revise with a correct re-observation. Only the full system should recover at a high rate. The second is the tax curve: accuracy at turn $k$ minus isolated accuracy of the same question, plotted against $k$ up to 10. Its slope should flatten for the full system relative to every baseline. The third is a turn-order permutation test, following SegLLM's finding that swapping past rounds breaks models. Because the ledger has no order, it should be invariant to permuting independent earlier turns, whereas transcript-based systems may not be. If the full system wins on the tax curve but not on the injected misread test, persistence helps but correction does not, and we will narrow the claim accordingly.