Gain-Scheduled Feedback Steering Against Text Bias in Vision-Language Models

Vision-language models often answer a question about an image by trusting the words around the image instead of the image itself. If a chart's title contains the answer, accuracy can reach 79%. If the title is removed or made misleading, accuracy falls to the 30% range (Plot Twist). When a caption contradicts the picture, most models side with the caption (Words or Vision). One cheap way to correct behavior at inference time is activation steering: adding a learned direction to the model's hidden states while it runs. Recent work recast steering as a feedback controller and added the integral and derivative terms of a classic PID controller (Activation Steering with a Feedback Controller). That work, like every multimodal steering method in this literature, sets its strength knobs by hand. Its authors say outright that automated tuning is still open.

We propose to treat the strength knobs of a steering controller the way control engineers treat the gains of a physical plant. The gains become the output of a sample-efficient search under an explicit capability constraint. They are also scheduled on a measured operating condition: how strongly the text in the prompt conflicts with the image. The question is narrow and testable. Is a closed-loop steering controller, with gains chosen automatically and adjusted to the conflict level, better at making a vision-language model rely on visual evidence than a simple steering vector given the same tuning budget? And does that fix transfer across task domains without hurting general visual-math accuracy?

Core Claim

We claim two things, each separable by ablation. First, much of the apparent difference between steering methods in multimodal settings comes from tuning, so any comparison must give every method the same automated tuning budget. Second, once tuning is equalized, two ingredients should still give gains that a single tuned strength cannot match, and those gains should be largest when the method is tuned on one domain and tested on another. The first ingredient is feedback: integral action that removes the residual bias a proportional push leaves behind. The second is gain scheduling: weak gains when text and image agree, strong gains when they conflict.

The physical-control literature motivates the transfer prediction. A cost weighting tuned under one operating condition fails under another (Cost Adaptation for Robust Decentralized Swarm Behaviour). Vehicle controllers depend on gain schedules indexed by speed and road geometry (Lateral control via Dual Heuristic Programming). In our setting, the operating condition is the degree of cross-modal conflict.

We are deliberately not claiming to close the multimodal reasoning gap. Two papers constrain the scope here. The fine-grained analysis on MATH-Vision attributes 42.2% of GPT-4V's errors to reasoning after correct perception and 31.9% to recognition (MATH-Vision). GeoSQA shows that models still fail when diagrams are replaced by human-written text (GeoSQA). A steering direction that pushes the model toward visual evidence cannot repair failures that persist when the visual evidence is given perfectly as text. We therefore target the share of failures caused by text shortcuts and modality preference. We treat general visual-math accuracy as a quantity to protect, and measure whether it improves rather than assume it will.

The Controller

We use the dynamical-system view of Nguyen et al., in which the hidden state $x_\ell$ at layer $\ell$ of the language backbone evolves as $x_{\ell+1} = f_\ell(x_\ell) + u_\ell + w_\ell$. Here $f_\ell$ is the layer, $u_\ell$ is our intervention, and $w_\ell$ is an internal disturbance. In our setting the disturbance is the text prior pulling the representation away from what the image supports.

The target $r$ is built from contrastive pairs: the same image paired with matching text versus contradicting text, following the corruption construction of Deng et al. This removes the need for an external object detector, which DMAS requires (Dynamic Multimodal Activation Steering). The error $e_\ell$ is measured only inside a low-dimensional subspace spanned by the contrastive directions, so the controller does not touch the rest of the state.

A small linear probe on a middle layer outputs a conflict score $c \in [0,1]$ in the same forward pass. This is the same kind of cheap detector Ferrando et al. use to decide when to steer (Dynamically Scaled Activation Steering). The control law is

$$u_\ell = K_p(c)\, e_\ell + K_i(c) \sum_{j=\ell_0}^{\ell} e_j + K_d(c)\,(e_\ell - e_{\ell-1}), \qquad K_\bullet(c) = a_\bullet + b_\bullet\, c,$$

where $\ell_0$ is the first steered layer. The pair $a_\bullet, b_\bullet$ gives the baseline gain and its slope in conflict for each of the three terms.

We place the controller in the language model's residual stream, not inside the vision encoder. This choice is forced by ARXSA's negative result: attention-level feedback breaks on Vision Transformers because their attention matrices are nearly singular (ARXSA).

All tunable quantities form a vector $\theta = (a_p, b_p, a_i, b_i, a_d, b_d, \ell_0)$. We choose them by solving

$$\max_\theta \; \mathrm{Acc}_{\text{conflict}}(\theta) \quad \text{subject to} \quad \mathrm{Acc}_{\text{clean}}(\theta) \ge \mathrm{Acc}_{\text{clean}}(0) - \delta .$$

Here $\mathrm{Acc}_{\text{conflict}}$ is accuracy on a tuning split of text-image conflict items. $\mathrm{Acc}_{\text{clean}}$ is accuracy on a held-out clean split drawn from MathVista. $\theta = 0$ is the unsteered model, and $\delta$ is a fixed tolerance of one accuracy point.

We solve this with constrained Gaussian-process Bayesian optimization, which fits a probabilistic model of score as a function of $\theta$ and picks the next setting to try. The method comes from the entropy-search approach that tuned LQR weights on hardware within a handful of trials (Automatic LQR tuning). The framing follows Ringrose's observation that automating tuning moves the engineering effort into writing the evaluation objective (Automatically Tuning Control Systems for Simulated Legged Robots). Our objective is that constraint.

Research Program

We will build the contrastive target and the conflict probe for three open models that appear in this literature: LLaVA-1.5, LLaVA-NeXT-7B, and Qwen2-VL. Each tuning run gets a fixed budget of 60 objective evaluations on a 500-item tuning split. Each evaluation is a full pass over the split, so cost is measured in forward passes and is identical across methods.

The analysis has three parts. First, a factorial comparison crosses two controllers with two tuning methods: a proportional-only controller (plain activation addition) versus PID, each tuned by the manual stability-first search of Nguyen et al. versus by Bayesian optimization. This separates the effect of the controller from the effect of tuning.

Second, a scheduling ablation compares fixed gains against conflict-scheduled gains, both tuned by Bayesian optimization.

Third, a transfer study tunes on one domain of the Deng et al. suite and tests on the others and on Chart101. This is the condition where the swarm and vehicle literature predicts that fixed gains break. We will also record the per-layer error trajectories $e_\ell$ to check the mechanism: whether integral action removes the plateau that a proportional controller leaves, as the PID steering analysis predicts under its local-linearity assumption, or whether that assumption fails in multimodal backbones.

Baselines

The naive reference point is the unsteered model with standard prompting. On MathVista, open models of this class score between 20% and 26%, against a frequent-guess baseline of 26.3% (MathVista). On the Deng et al. corrupted-text condition, base LLaVA-NeXT-7B reaches 28.69%. Every steered method must beat these numbers on the conflict sets while staying within $\delta$ of them on clean data.

The strongest existing method on this exact question is the supervised fine-tuning fix of Deng et al. It fine-tunes on 1,000 synthetic contrastive samples and raises LLaVA-NeXT-7B's corrupted-text accuracy from 28.69% to 71.25% (Words or Vision). We will reproduce it on the same 1,000-sample budget, reusing those samples to build our contrastive target, so both methods see the same data. Training-free steering reaching 71.25% in-domain would be a strong result, and we do not assume it. Our method counts as beating fine-tuning if it reaches within five points of it in-domain, matches or exceeds it in the cross-domain transfer cells, and loses less clean MathVista and MME accuracy.

The strongest training-free competitor is open-loop multimodal steering in the style of DMAS, which reports a +94.66 MME gain on LLaVA-v1.5 with hand-set strengths. We will run it with its strengths retuned by the same 60-evaluation Bayesian optimization. Second is DSAS-style probe-scaled proportional steering, which shares our conflict probe but uses no integral or derivative terms; it also gets the same budget. Third is PID steering with manually tuned gains, exactly as published.

The isolating ablations are the cells of the factorial and the scheduling comparison. Beating the Bayesian-optimized proportional controller isolates feedback. Beating fixed-gain Bayesian-optimized PID isolates scheduling.

Why not just…?

Why not just tune a single steering strength with Bayesian optimization and skip the PID machinery? This is the most serious threat, and the factorial design exists to settle it. If Bayesian-optimized activation addition matches Bayesian-optimized scheduled PID in-domain and in transfer, then the contribution reduces to "tune your steering automatically under a capability constraint." We would report that as the finding. The theory in Activation Steering with a Feedback Controller predicts a residual steady-state error for proportional control. DMAS reports that raising strength to compensate collapses general capability. Together these suggest the constraint will bind for the proportional controller before it closes the conflict gap. That suggestion is a hypothesis for the experiment, not a conclusion.

Why not just fine-tune, as Deng et al. did? Their result is the number to beat, and in-domain it may well win. The case for a controller rests on two measurable points: it can be retuned per deployment without touching weights, and scheduling should hold up better when the domain shifts. Fine-tuning large multimodal models also risks forgetting and loss of fluency outside the training distribution, as noted in DCoT. The transfer cells and the clean-accuracy constraint test both points directly. If fine-tuning wins on both, the controller has no practical case.

Why not just reorder the prompt or instruct the model to trust the image? Deng et al. found that placing text before the image inflates text preference, so putting the image first plus an explicit instruction is a nearly free baseline. We include it at zero tuning cost and also stack it under every steering method, so any steering gain is measured on top of it. Chain-of-thought prompting is a weaker candidate: MATH-Vision reports that it lowered performance for some multimodal models. We still run it once as a reference.

Proposed Experiments

The decisive checks are as follows. On the Deng et al. suite we report Text Preference Ratio and corrupted-text accuracy. On Chart101 we report accuracy with answer-bearing, removed, and misleading titles; the published spread is up to 79% versus the 30% range. Clean MathVista and MATH-Vision accuracy serve as the protected constraint. For MATH-Vision the reference is 16.97% for the strongest open model, InternVL-Chat-V1-2-Plus. MME measures general capability.

The proposal succeeds if three things hold. Scheduled PID with Bayesian-optimized gains closes more of the Chart101 title gap and the corrupted-text gap than Bayesian-optimized proportional steering, at equal budget and with clean accuracy within $\delta$. The advantage grows in cross-domain transfer cells. And the per-layer error trajectories show the plateau-removal mechanism rather than a larger push in disguise.

Three outcomes would falsify the main claim, each with its own interpretation. If clean MathVista accuracy cannot be held, the error subspace is leaking into reasoning. If manual and automated tuning give the same ranking of methods, tuning was not the confound we suspected. If Bayesian-optimized proportional steering ties scheduled PID, feedback adds nothing here.

The secondary question is whether reducing text reliance moves MathVista and MATH-Vision accuracy at all. The GeoSQA and MATH-Vision error analyses suggest any such gain will be small. Reporting its size honestly is part of the contribution.