Choosing the Noise Level That Survives Chart Restyling

Vision-language models, which pair an image encoder with a language model to answer questions about pictures, fail on surprisingly basic chart questions. They misread axis limits and point coordinates, and their accuracy drops when a plot gains grid lines or switches plotting library. Separately, a diffusion model is a generator trained to turn random noise into images by removing noise step by step. Its internal activations make strong visual features, but only if you feed it an image corrupted to the right noise level, the "timestep." We propose to pick that timestep by a single criterion: the features should not change when the same chart is restyled. We then feed features from the selected timesteps to a vision-language model as extra visual input, and test whether this repairs the elementary chart perception that current models lack.

The two-sentence version is this. The cosmetic changes that break vision-language models on charts (grid lines, marker shapes, plotting engines) can be used as "views" in an existing label-free measure of how much diffusion features mix stable content with noise-like clutter. Minimizing that measure over timesteps should select diffusion features that encode what the chart says rather than how it was drawn, and those features should make a vision-language model robust where the standard image encoders it normally uses are not.

Core Claim

Three findings motivate the claim. First, Plot Twist built Chart101, a set of elementary questions (axis minimum and maximum, coordinates read within 5% tolerance). It showed that models score worse on these questions than on complex chart reasoning. Accuracy fell from up to 79% when the title leaked the answer to around 30% without it, and PaLI-3 dropped from 29.5% to 21.3% when only grid lines were added. Second, the error analysis in MATH-Vision attributes 31.9% of GPT-4V failures to vision recognition. That leaves a large perception share that a better visual signal could address, even though reasoning errors (42.2%) remain the majority. Third, diffusion features beat standard self-supervised encoders on localization, and the effect depends on timestep. ConceptAttention takes a linear projection of the output of the attention layers inside a diffusion transformer and reaches 71.04 mIoU on ImageNet-Segmentation at an intermediate step, against 69.44 for DINO. Label-Efficient Semantic Segmentation with Diffusion Models shows that both very low and very high noise hurt.

The method comes from Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles. It defines the Invariant Contamination Ratio (ICR), a Fisher-discriminant-style split of features into a part that stays stable across views and a residual. That paper reports that the timestep minimizing ICR coincides with the timestep giving peak linear-probe accuracy. Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling explains why a single interior optimum should exist and shows that ensembling several timesteps helps slightly: 78.4% versus 77.6% linear-probe accuracy on MiniImageNet.

Our claim is that, on abstract graphics, the right "views" for ICR are restylings of the same underlying data. The ICR-minimizing timestep differs by primitive type: we expect low noise for tick text and exact positions, and intermediate noise for grouping marks and matching legend colors to series. A small bank of ICR-selected timesteps should then outperform any single fixed timestep, the noise-free input used by prior perception work, and standard contrastive or self-supervised encoders on style-perturbed chart perception.

We adapt ICR to charts as follows. For a feature map $f_{t,\ell}(x)$ taken from layer $\ell$ at noise level $t$ for chart image $x$, let $\{x_{i,1},\dots,x_{i,K}\}$ be $K$ renderings of the same data table $i$ under different styles. We score

$$ \mathrm{ICR}(t,\ell) = \frac{\operatorname{tr}\,\Sigma_{\text{within}}(t,\ell)}{\operatorname{tr}\,\Sigma_{\text{between}}(t,\ell)}, $$

where $\Sigma_{\text{within}}$ is the covariance of features across renderings of the same table and $\Sigma_{\text{between}}$ is the covariance of per-table mean features. A low score means the features track the data and ignore the drawing style. No question labels are needed. This is not the same as reweighting timesteps on a labeled validation set, which is what Diffusion Classifiers Understand Compositionality, but Conditions Apply does. That paper raised CLEVR binding from 63% to 98%, but it needs a full diffusion-loss evaluation for every candidate answer and cannot answer open-ended questions. We use timesteps only to choose a feature extractor, run in a single forward pass per timestep.

Research Program

The first stage is a diagnostic atlas. We render synthetic charts (bar, line, scatter) and simple geometry figures from known data tables, each under several styles: grid lines on and off, different marker shapes, two or more plotting engines, and recolored palettes. We pass each rendering through a multi-modal diffusion transformer of the kind used by ConceptAttention, Flux.1 or SD3.5. We take attention-output features at roughly ten timesteps and several depths, and compute ICR per primitive family. In parallel, we train linear probes for each primitive: tick-value regression, point-coordinate regression, mark-to-legend matching, and the sign of a trend. These probes test whether ICR minima predict probe optima on graphics, as they did on natural-image classification.

This stage also tests a known constraint. Diffusion Classifiers documents a "native language" effect: diffusion models degrade sharply on images unlike their generative training data. Charts may be exactly such images. The atlas will report this directly, as probe accuracy of diffusion features versus SigLIP and DINOv2 on real ChartQA charts compared with synthetic ones.

The second stage is integration. We keep a LLaVA-style model built on SigLIP fixed and add a second stream of visual tokens. Each token comes from diffusion features at $M \le 3$ ICR-selected timesteps, pooled to a fixed token budget and projected by a trained connector. A small question-conditioned gate assigns softmax weights over the $M$ timesteps. This is the dynamic, per-query selection that the file identifies as missing from fixed-$t$ methods. We keep the gate deliberately minimal, so that we can tell whether gains come from the gate or simply from timestep choice. We train the connector and the gate on the standard instruction-tuning data plus our rendered chart data. The diffusion transformer stays frozen.

The third stage is cost accounting. Each timestep costs one diffusion-transformer forward pass, not a sampling loop. We report wall-clock time and FLOPs relative to the SigLIP-only model. This matters because VPD chose the noise-free input $t=0$ precisely to reduce cost to one pass, and we need to show that the added passes pay for themselves.

This work is distinct from the two closest bridges in the file. DS-VLM attaches a diffusion model as a pixel-reconstruction loss during training, samples noise uniformly, and discards the diffusion model at inference. We keep diffusion features at inference and choose timesteps deliberately. The low-dimensional and ICR analyses also give reason to suspect that pixel-level reconstruction pulls features toward high-frequency clutter. Thinking Diffusion applies timestep penalties and guidance to the text side of diffusion language models such as LaViDa, while still using static SigLIP-type visual features. We change the visual side and leave the language model autoregressive, so the two approaches are complementary rather than competing.

Baselines

The naive reference point is the same LLaVA-style model with only its SigLIP encoder. It is fine-tuned on exactly the same instruction and chart data for the same number of steps, so any gain from data alone is credited to it.

The strongest non-diffusion competitor is a second stream of DINOv2 tokens in place of diffusion tokens. It uses the same connector architecture, token budget, and training schedule, with the learning rate tuned over the same grid. This is the alternative most likely to match us, because DINO is close to ConceptAttention on localization (69.44 versus 71.04 mIoU). Beating it requires a statistically significant gain on style-perturbed Chart101 accuracy across seeds.

The strongest training-time diffusion method is DS-VLM, re-implemented on the same base model and data. DS-VLM reports a 2.1% improvement on MMMU and a 2.4% improvement on MMBench with Vicuna-7B. Matching those general-benchmark deltas while exceeding it on chart perception would count as a win. On ChartQA, the reference human-question figure from ChartQA is about 30% for VisionTaPas and VL-T5. No reported ChartQA number was found on Lacuna for LaViDa. On MathVista, the strongest number in the file is 68.4% for LaCoT-Qwen-7B from Latent Chain-of-Thought for Visual Reasoning. That is a reasoning-side method on a stronger base model, so we treat it as an upper reference rather than a direct competitor.

Three diffusion ablations isolate the key ingredient, using identical token budgets. The first is VPD-style noise-free features at $t=0$. The second is ConceptAttention's fixed intermediate step. The third is a uniform concatenation of the same $M$ timesteps without ICR selection and without the gate. We also include a version with ICR selection but no gate. Our method has to beat the fixed-step variants for timestep selection to matter, and has to beat uniform concatenation for ICR to matter. If it only beats them with the gate, the gate becomes the contribution.

Why not just…?

Why not just fine-tune SigLIP with style augmentations? If the failure is fragility to restyling, augmenting the existing encoder with the same rendered perturbations is the cheapest fix, and it is a real threat. We run it as a baseline with identical rendered data and compute. If it closes the robustness gap on held-out styles, the diffusion stream is unnecessary and we will report that. Our bet is that augmentation helps on styles it has seen but not on unseen plotting engines. That is why the test splits use styles absent from training.

Why not just concatenate a few timesteps and skip selection? The low-dimensional analysis already shows ensembling beats a single best step on MiniImageNet, by 78.4% versus 77.6%. Uniform concatenation is therefore the ablation that could make ICR selection pointless. If it matches ICR selection at equal token budget, we conclude that selection does not matter for charts. The atlas still stands, but the method claim drops.

Why not just use OCR and code execution? Converting charts to tables and computing in Python, as in Synthesize Step-by-Step, is strong for numeric aggregation. ChartQA reports such pipelines falling to about 30% on human-written questions that refer to visual attributes like "the tallest green bar." We evaluate on that human-question subset and on Chart101 scatter coordinates, where text extraction does not apply.

Why not just prompt for step-by-step reasoning? MATH-Vision reports that chain-of-thought prompting lowers accuracy for some multimodal models. Plot Twist shows that apparent reasoning competence hides elementary perception failures. Chain-of-thought addresses a different stage of the problem, so we will apply it on top of every baseline equally rather than treat it as a substitute.

Proposed Experiments

The first decisive check is the atlas. On synthetic charts with held-out styles, does the ICR-minimizing timestep per primitive family fall within one grid step of the probe-optimal timestep? Are the optima for coordinate reading and for legend matching measurably different? If the optima coincide, a single fixed timestep suffices and the gate is dropped. If diffusion probes trail DINOv2 on real ChartQA charts, the domain-gap concern is confirmed and we stop before integration.

The second check is robustness. We evaluate all baselines on Chart101 and on a perturbed Chart101 with grid lines, marker swaps, and engine swaps. We report the gap between clean and perturbed accuracy, in the spirit of the PaLI-3 drop from 29.5% to 21.3%, and the title-ablated accuracy that exposes language shortcuts. Success means a smaller perturbation gap and higher title-removed accuracy than both the augmented-SigLIP and DINOv2 baselines.

The third check is transfer. We evaluate on ChartQA-H, MathVista, and the geometry and arithmetic subsets of MATH-Vision. Since reasoning errors dominate MATH-Vision, we expect gains there to be bounded by the 31.9% vision-error share. We predict improvements concentrated in perception-heavy categories, and we will verify this by re-labeling errors with the MATH-Vision error taxonomy. A gain spread evenly across reasoning categories would suggest that something other than perception is driving the improvement.