Training data for chart and diagram reasoning is increasingly built in loops: a model drafts an annotation (a bounding box, an extracted data table, a step-by-step rationale), a human corrects it, the model is retrained on the corrections, and the next round's drafts improve. These loops assume the humans are honest. Crowdsourcing security research shows that strategic annotators can build up "trust capital" on easy items and spend it on targeted ones. This proposal argues that iterative loops are more exposed to that attack than one-shot crowdsourcing, for two reasons. In multimodal reasoning data, the easy-to-verify part of an annotation (reading a value off an axis) is separable from the hard-to-verify part (the multi-step deduction). And a poisoned correction that gets retrained into the drafting model comes back as next round's draft, where reviewers who tend to accept drafts will wave it through.
We propose to measure this compounding effect and to build a defense against it. The defense has two parts. Reputation credit is earned only from actions that carry information: agreeing with a confident, correct draft earns almost nothing, so there is no cheap way to recharge. Perception and reasoning annotations are also split across independent workers, and each reasoning program is executed against another worker's extracted table, so a poisoned rationale cannot be made internally consistent without collusion across stages. The testbed is chart question answering, where the gap between seeing and reasoning is large and well documented.
Core Claim
In a multi-round, model-in-the-loop pipeline for chart reasoning data, a camouflaged annotator who is accurate on perception fields and selectively wrong on reasoning fields will degrade the downstream model more per poisoned item than the same attacker in a static label pool, because retraining recycles the poison into drafts. Standard reliability estimation cannot see this. It scores workers by agreement, and agreement is dominated by easy perceptual items.
A reputation rule that weights credit by how likely the draft was to be wrong, tracks perception and reasoning reliability separately, and cross-checks reasoning programs against independently produced tables should cut this damage substantially. It should do so at a much lower annotation and compute cost than auditing everything after the fact. Each half of the claim can fail on its own: the amplification may not exist, or simpler defenses may already contain it. The experiments are designed to tell these cases apart.
Grounding in Prior Work
Three findings motivate the threat model. First, Black-Box Data Poisoning Attacks on Crowdsourcing introduces SubPac. It poisons label aggregation without seeing the aggregator, by optimizing against a stand-in aggregator such as Dawid-Skene (a model that estimates each worker's error rates jointly with the true labels). It targets ambiguous items, and its attackers reach estimated reliability of 0.75–0.90 against an honest average near 0.80. SubPac works on static discrete-label pools (Temp, Rte, Sentiment, ER) and does not consider structured visual annotations or drafting loops. Those are exactly what we add.
Second, Trust Models and Con-Man Agents proves that reputation models that discount old history fall into a "memory trap": an attacker can defect and recover indefinitely at constant cost. The same paper proposes AER, which makes each successive recovery more expensive, growing from 10 to nearly 100 cooperative interactions over five cycles. That analysis uses abstract agents with scalar payoffs. We reuse AER's escalating penalty as one component and test it in a setting where "cooperation" is cheap to fake.
Third, the multimodal reasoning literature explains why fakeable cooperation is easy here. MATH-Vision attributes 31.9% of model errors to visual recognition and 42.2% to reasoning that fails after correct perception. Plot Twist shows that models lean on chart titles, reaching 79% accuracy when the title gives away the answer and dropping to the 30% range without it. Perception fields are therefore where an honest-looking attacker earns credit. Reasoning fields are where errors are hard for models, and likely for tired reviewers, to catch.
The method draws on two papers. Synthesize Step-by-Step represents chart rationales as atomic sub-questions paired with executable Python run over a DePLOT-extracted table. That structure makes the perception–reasoning split explicit and machine-checkable. We reuse the format but put humans in the loop, which that paper did not. InfoDet supplies the loop design: eight rounds of detector drafts corrected by experts over 11,264 real infographics. Its authors note that reviewers default to accepting drafts at ambiguous boundaries. We treat that anchoring effect as the channel through which poison compounds. InfoDet assumes its annotators are reliable, and we drop that assumption.
Two papers constrain the design. ActiveAED shows that iterative, human-guided error detection beats one-pass ranking. It needed roughly 1,000 GPU hours across eight text datasets, so any defense that requires repeated full retraining of a multimodal model is impractical. Provably manipulation-resistant reputation systems gives strong group-regret guarantees via semidefinite programming. It is computationally heavy and assumes sequential transactions, whereas annotation rounds are batched and concurrent. We therefore aim for a cheap heuristic rule with empirical, not provable, robustness, and we say so plainly.
Research Program
Testbed. We build a controlled annotation loop over ChartQA charts (ChartQA). Each item consists of a chart and a human-authored question, and its annotation has three layers:
- a perception layer: the extracted data table, plus boxes for referenced marks in the style of InfoDet;
- a reasoning layer: an executable rationale in the Synthesize Step-by-Step format;
- the final answer.
A drafting model proposes all three layers. Annotators accept or edit each layer, and the drafting model is fine-tuned on accepted data between rounds. We use four to eight rounds, matching InfoDet's scale of iteration. To stay within a modest budget, retraining is parameter-efficient adaptation of a mid-sized open chart model, not full retraining. This responds directly to ActiveAED's cost finding.
Annotator population. Honest annotators are simulated from real ChartQA human-split answers, plus a calibrated rate of draft-acceptance errors. The acceptance error rate is higher on ambiguous items, to mimic the anchoring InfoDet reports. A small paid study with real annotators then measures actual acceptance rates on wrong drafts, so the simulation is not purely assumed.
Attackers come in three increasingly strong forms:
- a strike-and-recharge attacker that is accurate on perception layers and poisons reasoning programs on a targeted question family, such as percentage-change questions;
- a SubPac-style attacker that optimizes its targeting against a stand-in Dawid-Skene aggregator;
- a loop-aware attacker that plants poisons designed to be redrafted. These are consistent, plausible errors that the retrained model will reproduce.
Defense. The defense has three pieces that can be ablated separately.
- Informativeness-weighted credit. When annotator $a$ acts on layer $\ell$ of item $i$ and the action is later confirmed, their stage-specific trust $T_{a,\ell}$ grows by an amount that depends on how informative the action was:
$$\Delta T_{a,\ell} = u_{i,\ell}\, c_{i,\ell} - \lambda_{a,\ell}\,(1 - c_{i,\ell})$$
Here $u_{i,\ell} \in [0,1]$ is the drafting model's estimated probability that its own draft of layer $\ell$ is wrong, taken from ensemble disagreement over the last round's adapters. The term $c_{i,\ell} \in \{0,1\}$ records whether the annotator's action was later confirmed. $\lambda_{a,\ell}$ is an AER-style penalty that doubles with each confirmed defection by that annotator on that layer. The effect is that accepting a confident correct draft is worth nearly nothing, which closes the cheap recharge path.
- Stage separation. Perception and reasoning layers of the same item go to different annotators, and $T_{a,\ell}$ is tracked per layer. Trust earned reading axes does not carry over to rationales.
- Cross-execution. Every submitted reasoning program is run against a perception table produced by a different annotator. An answer mismatch sends the item to a high-trust adjudicator. A lone attacker cannot then make a wrong rationale self-consistent unless the table is also wrong, and that requires collusion across stages.
Analysis. We will measure how the attacker's damage per poisoned item grows across rounds. Our working hypothesis is that it grows roughly with the product of the redraft rate (how often the retrained model reproduces a planted error) and the honest acceptance rate on wrong drafts, since each round's surviving poison is the previous round's poison that was both redrafted and accepted. We will estimate both rates from data rather than assume them.
Baselines
The naive reference point is the loop exactly as InfoDet runs it: drafts, human edits, retraining, and no worker weighting at all. This gives the undefended damage curve, and every defense is reported as a fraction of the damage it removes.
The strongest standard aggregation competitor is Dawid-Skene worker reliability estimation applied per layer, with low-reliability workers down-weighted. It is the stand-in model SubPac attacks and the default in crowdsourcing practice, so it is the natural thing a practitioner would deploy. Black-Box Data Poisoning Attacks on Crowdsourcing reports that camouflaged attackers reach reliability scores of 0.75–0.90 against honest workers near 0.80, and that SubPac's attack success is up to 5.7 times that of the TIA baseline. Those numbers come from text classification. No reported result was found on Lacuna for Dawid-Skene, or any aggregation defense, inside a multimodal model-in-the-loop annotation pipeline. We beat this baseline if, at equal adjudication budget, the targeted-family accuracy loss is at most half that under Dawid-Skene, with no more than one point lost on untargeted ChartQA human-split accuracy.
The second strong competitor is AER's escalating penalty (Trust Models and Con-Man Agents) applied to a single trust score per worker that counts every confirmed action equally. This isolates whether our gain comes from escalation, which is borrowed, or from informativeness weighting and stage separation, which are new.
The third is ActiveAED (ActiveAED) run as a post-hoc auditor on the final dataset with the same total human review budget. The file reports it lifting average precision on SI-Flights from 80.3% to 86.6%, a text result with no chart counterpart on Lacuna.
All defenses get identical charts, annotator pools, attacker budgets, number of rounds, adjudication calls, and adapter-training compute. Hyperparameters such as trust thresholds are tuned on a held-out set of attack-free rounds only.
The ablations remove informativeness weighting (setting $u = 1$), stage separation, and cross-execution one at a time. If removing informativeness weighting costs nothing, the core claim about cheap recharge is wrong.
Why not just…?
Why not just seed gold questions with known answers? This is the most serious alternative, and we treat it as a real threat. Gold items drawn from easy charts are easy for an attacker to spot and answer honestly. That is the same camouflage SubPac exploits on easy consensus items. Gold drawn from hard reasoning items is expensive to author and runs out over many rounds. We will run a gold baseline with hard-item gold matched to our adjudication budget, at three gold rates. If it matches our defense, the contribution narrows to the amplification measurement and a recommendation to put gold on reasoning layers. We would report that outcome as it stands.
Why not just execute the rationale and reject programs whose answer is wrong? Execution checks that the program is internally consistent, not that it is true. A poisoner who also controls the table can make a wrong program run cleanly. In the purely synthetic setting, Synthesize Step-by-Step already leaves hallucinated logic and DePLOT extraction errors unguarded. Our cross-execution ablation, which executes against the annotator's own table instead of an independent one, directly measures how much of the gain depends on independence.
Why not just drop humans and use synthetic data? ChartQA reports roughly 61% on machine-generated questions against roughly 30% on human-authored ones. Synthetic rationales lifted the human split only to 53.9% in Synthesize Step-by-Step. Human-authored reasoning remains the scarce input. As a reference arm, we include a model trained on our pipeline's charts with fully automatic drafts and no human edits.
Why not just audit once at the end? The ActiveAED baseline covers this. Our hypothesis is that end-of-loop auditing is too late, because by then the poison has been redrafted and accepted across many items, which inflates how much needs auditing.
Proposed Experiments
Experiment 1: does amplification exist? We plant a fixed number of reasoning-layer poisons in round one, then:
- track how many poisoned items exist after each round;
- estimate the redraft and acceptance rates;
- measure accuracy on the targeted question family in ChartQA's human split.
We compare against the same poisons placed in a single-round static pool. The decisive check is whether loop damage per planted item exceeds static damage by round four. If it does not, the threat model is overstated and the proposal reduces to a static-crowdsourcing study.
Experiment 2: defense comparison. We run all baselines against the three attacker types at attacker fractions from 5% to 30%. This upper bound follows SubPac's finding that it stays potent with access to only 30% of honest labels. Metrics are:
- targeted-family accuracy loss;
- untargeted human-split accuracy, alongside the published 53.9% human-split result for LAMENDA-trained models (Synthesize Step-by-Step), cited only as context;
- average precision for flagging attackers;
- "laundering cost": how many honest-looking actions an attacker needs per successful poison.
The decisive check is whether informativeness weighting raises laundering cost by an order of magnitude over Dawid-Skene and AER, as the design intends.
Experiment 3: transfer and robustness of the trained model. Models trained on data from each defense are evaluated on:
- Chart101 from Plot Twist, including its title-removed and grid-line variants, to check whether poisoned reasoning increases reliance on titles;
- the chart and statistics subsets of MathVista, where GPT-4V reaches 49.9% against 60.3% for humans.
We will also replicate Experiment 2 at smaller scale on the scientific figures of VeriSciQA, for which no reported result was found on Lacuna. This tests whether the stage split carries over beyond simple bar and line charts.
Experiment 4: real annotators. A paid study measures honest annotators' acceptance rate on deliberately wrong reasoning drafts. If that rate is near zero, the anchoring channel, and with it the amplification, is weak in practice, and we report that as the main finding.
References
- Black-Box Data Poisoning Attacks on Crowdsourcing
- Trust Models and Con-Man Agents: From Mathematical to Empirical Analysis
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- InfoDet: A Dataset for Infographic Element Detection
- ActiveAED: A Human in the Loop Improves Annotation Error Detection
- Provably manipulation-resistant reputation systems
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts