When a vision-language model is fine-tuned on step-by-step solutions to chart and diagram problems, two things can happen at once. It gets better at deduction, and it also learns to answer from the question text and chart title instead of from the image. Benchmarks show both pressures are real. On MATH-Vision, 42.2% of GPT-4V's errors are reasoning errors made after correct perception, while 31.9% are perception errors (MATH-Vision). On Chart101, models read simple chart values worse than they answer complex chart questions, and accuracy falls from about 79% to the 30% range once the answer is removed from the title (Plot Twist). This proposal asks whether these two effects of fine-tuning can be pulled apart inside the weight update itself. If they can, we can keep the reasoning gain and remove the shortcut.
The tool is a low-rank adapter (LoRA, a small trainable update $\Delta W = BA$ added to frozen weights). We read the adapter as a sum of rank-1 pieces, $\Delta W = \sum_i b_i a_i^\top$. Each piece produces one scalar per token, so it can be audited one at a time. For each piece we measure whether its activity depends on the actual image. We do this by re-running the model on a counterfactual chart: the same chart re-rendered with different data values but an unchanged title and question. Pieces that fire the same way on the real and counterfactual image are acting on language alone. The core claim is that, in a multimodal reasoning fine-tune, the rank-1 pieces that raise deduction accuracy and the pieces that lower grounded chart reading are largely different pieces. Removing or down-weighting image-insensitive pieces should then restore elementary perception without giving up the reasoning gain. This can be tested directly, and the test does not depend on any new architecture working.
What exists and what is missing
Three lines of work treat rank-1 components as separate experts. MoDE shares the down-projection and routes among candidate up-projection vectors per rank. It reaches 60.00 ROUGE-L on 756 Supernatural Instructions tasks against 56.11 for LoRA, but it is text-only. MixLoRA selects factors of $A$ and $B$ conditionally for multimodal instruction tuning and scores 1509.61 on MME against 1393.67 for LoRA, but it is never tested on multi-step visual math. Fa et al. and KSS-MoE route rank-1 or subspace experts and add orthogonality losses to prevent forgetting across a sequence of classification or VQA tasks. All four route or regularize by task or input identity. None asks whether an individual rank-1 piece actually uses the image. That question is ours.
Rank-1 LoRAs Encode Interpretable Reasoning Signals is the closest motivation. It shows that a single rank-1 adapter in a 32B text model recovers 73–90% of full fine-tuning's reasoning gains, that its activations track interpretable reasoning markers, and that MLP gate_proj adapters carry most of the effect. It is text-only, and its steering interventions were inconclusive. We borrow its placement (MLP gate projections) and its premise that rank-1 units are readable. We replace activation steering with a weaker but more reliable intervention: deleting a piece and measuring the outcome.
The constraining paper is Position: Reasoning After Perception Means Reasoning Without Vision. It argues that once visual information is collapsed into language tokens, language-side tuning cannot recover it, and it reports near-zero accuracy on the Turing Eye Test. We take this seriously. If it is right, the language model may contain no image-sensitive rank-1 pieces worth protecting, because grounding is lost upstream. So we also place a rank-1 pool in the vision-to-language projector. The location where grounded pieces appear, if they appear at all, is a result in its own right.
Core Claim and Mechanism
We train a pool of $r$ rank-1 pieces with token-level sigmoid gates $g_i(x_t)$ on multimodal reasoning traces. For piece $i$, let $c_i(x_t) = g_i(x_t)\, a_i^\top x_t$ be its scalar contribution at token $t$. Let $\tilde{x}_t$ be the same token's hidden state when the image is replaced by its counterfactual re-rendering. The grounding score is
$$G_i = \frac{\mathbb{E}_t\,|c_i(x_t) - c_i(\tilde{x}_t)|}{\mathbb{E}_t\,|c_i(x_t)|},$$
with the expectation taken over answer and intermediate-value tokens of chart problems. $G_i$ near zero means the piece contributes the same amount regardless of what the chart shows.
Two operations use this score. Post-hoc audit deletes pieces in increasing order of $G_i$ and traces the resulting curve of perception accuracy against reasoning accuracy. Grounding-aware training adds a penalty $\lambda \sum_i (1 - \hat{G}_i)\, \mathbb{E}_t |c_i(x_t)|$ on perception-critical tokens. Here $\hat{G}_i$ is a running estimate of $G_i$ computed on counterfactual pairs, and $\lambda$ is the penalty weight. The penalty discourages image-blind pieces from carrying the answer to "what is this value" sub-steps. It leaves them free to carry arithmetic and discourse steps, where image-blindness is appropriate. This split is something a monolithic adapter cannot express.
Research Program
The base model is LLaVA-1.5-7B. It is the backbone used by KSS-MoE, and MathVista reports LLaVA-class models at 20–26% (MathVista), so there is room to move. A replication on a second open backbone checks generality.
Training data has three parts. The first is ChartQA training charts with distilled step-by-step rationales, following the recipe of Chart-based Reasoning. The second is template-generated chart sub-questions in the style of Synthesize Step-by-Step. Because these are synthetic, every chart can be re-rendered with edited values to produce exact counterfactual pairs. The third is geometry training problems from MathVista's source datasets, with MathVista test items removed. For geometry, counterfactuals are built by editing labelled lengths or angles in the image only.
The first phase is diagnostic. We fine-tune with a plain monolithic LoRA and confirm the Plot Twist-style trade-off: reasoning accuracy rises while elementary reading and title-swap robustness fall. Without this trade-off the rest of the program has no target. The second phase trains the gated rank-1 pool at matched parameter count. It computes $G_i$ for every piece, in both the LLM MLPs and the projector, and runs the deletion-curve audit. The third phase trains with the grounding penalty. A further analysis checks whether low-$G_i$ pieces are the ones that fire on reasoning markers, using the activation-labelling approach of Ward et al., and whether the separation is sharper in the projector or in the LLM.
Baselines
The naive reference is the frozen base model evaluated zero-shot on every benchmark, alongside the frequent-guess heuristic of 26.3% on MathVista and random chance of about 5.86% on MATH-Vision, both from the benchmark papers.
The first trained reference is a monolithic LoRA with the same trainable-parameter count, the same data and steps, and a learning-rate sweep of identical size. It defines the trade-off we are trying to escape.
The single-rank-1 adapter of Ward et al., trained on the same multimodal traces, tests whether a minimal update already avoids shortcut learning. Its smallness might simply leave perception intact.
The strongest existing rank-wise method on this question is MixLoRA. It is multimodal, routes individual factors, and was designed against interference. It reports 1509.61 on MME versus 1393.67 for LoRA. We found no reported result on Lacuna for MixLoRA on MathVista, MATH-Vision, or Chart101, so we will run it ourselves. It gets matched parameters, the same data, and the same tuning budget, with its expert count chosen by the same sweep.
The orthogonality alternative is the activation-guided orthogonal loss of Fa et al.. We repurpose it by treating perception-only training as the "old task" whose critical pieces must be protected. It tests whether protecting perception through geometry works as well as protecting it through measured image-dependence.
Two ablations isolate the key ingredient. One deletes pieces at random to the same budget. The other deletes pieces in order of smallest weight norm. If deletion by $G_i$ does not beat both by a clear margin on the perception–reasoning frontier, the grounding score adds nothing.
Beating the field means one of two outcomes against the best competitor at equal compute: higher Chart101 collective accuracy and title-swap accuracy at no loss on MathVista or MATH-Vision, or higher reasoning accuracy at equal perception. Our working threshold is a gap larger than the seed-to-seed spread over three seeds. For upper references we report numbers from the file, without claiming to match them: 81.3% on ChartQA for ChartPaLI-5B with program execution (Chart-based Reasoning), and 30.39% on MATH-Vision for GPT-4o.
Why not just…?
Why not just train on the counterfactual pairs directly, as data augmentation? This is the most serious threat, and we treat it as a co-equal arm rather than a strawman. A monolithic LoRA trained with the same counterfactual pairs as extra supervised examples may close most of the perception gap on its own. We expect augmentation to help on the synthetic chart distribution it was built from, and to transfer less well to the real-chart Chart101 split and the title and gridline perturbations, because it teaches correct answers without constraining which parameters carry them. If augmentation matches grounding-aware training on held-out real charts, we will report that the mechanism is unnecessary for accuracy. The audit would still have value as an explanation of where shortcuts live. We will also test augmentation combined with the penalty.
Why not just route visual tokens and text tokens to separate adapters? Modality routing is cheap, and it resembles question-level routing in Octavius. The shortcut, however, operates on text tokens: the answer token is generated from the title rather than the image, and in both cases it is a text token. A modality split cannot tell a grounded answer token from an ungrounded one. $G_i$ makes that distinction by construction. We include a modality-split LoRA at matched parameters as an additional baseline to confirm this empirically.
Why not just de-render the chart to a table and execute code? Chart-based Reasoning and Synthesize Step-by-Step show that this works very well on ChartQA. The file also notes that it breaks on geometric and non-tabular images, which make up much of MATH-Vision and GAOKAO-MM (GAOKAO-MM). We therefore expect the tool route to win on ChartQA and to lose on geometry. Our MATH-Vision geometry subjects, where GPT-4o reaches roughly 18% on transformation geometry, are where this is decided.
Proposed Experiments and Decisive Checks
The perception axis uses Chart101 elementary questions, reported as both per-question and per-chart collective accuracy, plus our re-run of the Plot Twist perturbations: title removal or swap, and added gridlines. The reasoning axis uses MathVista testmini, MATH-Vision by subject, and the math and physics portion of GAOKAO-MM. ChartQA, split into human-written and augmented questions, sits between the two axes. Every method is plotted on the perception–reasoning plane across three seeds.
The first decisive check is separability. After the monolithic-LoRA trade-off is confirmed, deleting the lowest-$G_i$ fifth of pieces from the trained pool should recover at least half of the lost Chart101 accuracy. MathVista accuracy should fall by less than the seed spread, and random or norm-based deletion of the same budget should not achieve this. If no deletion order does better than random, the claim that reasoning and shortcuts occupy different rank-1 directions is false, and we will report it as such.
The second check is location. If grounded, high-$G_i$ pieces appear only in the projector pool and not in the LLM MLPs, that supports the Position paper's account of early collapse and favors placing adapters upstream. If they appear in the LLM as well, language-side tuning can still route visual evidence, which is an answer to the tension between that paper and Ward et al.
The third check is the head-to-head against counterfactual augmentation on held-out real charts and geometry. This determines whether the grounding penalty improves accuracy, or whether its value is limited to explaining where shortcut learning lives inside the adapter.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts
- Multimodal Instruction Tuning with Conditional Mixture of LoRA
- Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA
- KSS-MoE: Knowledge Space Synergy Framework in Mixture of Experts for Continual Visual Instruction Tuning
- Rank-1 LoRAs Encode Interpretable Reasoning Signals
- Position: Reasoning After Perception Means Reasoning Without Vision
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
- GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation