AROMA: Adaptive Representation-Oriented Multimodal Actuation for Vision-Language Model Control

Zhaoning ZhangarXiv 2026 · arXiv

TL;DR

AROMA (Adaptive Representation-Oriented Multimodal Actuation) is a new framework that transforms mechanistic interpretability from a diagnostic tool into a precise control system for frozen Vision-Language Models (VLMs). By factorizing the control process into distinct stages—where, how, and when to intervene—it significantly improves a model's ability to count objects, achieving an 81% success rate in reaching target numerals where traditional steering methods fail.

What Problem Does This Paper Address?

Even state-of-the-art VLMs are often "4-knowers": they can reliably count a few objects but struggle as numbers increase. While researchers can often locate specific "attention heads" responsible for counting, simply "steering" these heads (adding a vector to their activations) frequently fails to produce reliable results.

The authors identify several reasons for this "mechanism-to-control gap":

  1. Sensitivity vs. Energy: The internal directions that most influence counting behavior often contain very little of the model's actual activation energy. Naive steering in these directions is too "weak" to change the output.
  2. Finite Scale Failure: A steering direction that works for a tiny change might point the wrong way when you need a large change (e.g., moving a count from 2 to 8).
  3. Target Uncertainty: We don't always know which numeral the model should have predicted, making it hard to know where to steer.
  4. Collateral Damage: Intervening on every example can break predictions that were already correct.

Methods

AROMA addresses these gaps by factorizing the control stack into five key decisions:

  • Where: It identifies "causal actuators"—specific cross-attention heads—only after bidirectional tests prove they actually move the count.
  • Which Geometry: It introduces Magnitude-Normalized Causal Subspace Actuation (MN-CSA). This technique scales low-rank edits to match the energy of full-head interventions, overcoming the sensitivity-energy mismatch.
  • How: It uses Minimal-Effort Actuation (MEA), an iterative process that re-linearizes the "target-competitor margin" at every step. This allows the system to navigate the model's internal landscape to reach a specific numeral target with minimal displacement.
  • Which Target: It employs a "runner-up" rule—pursuing the second-most-likely numeral when the model is uncertain—avoiding the need for ground-truth labels at inference time.
  • When: A learned "gate" decides per-example whether to intervene at all, preserving correct baseline predictions.
The AROMA control stack factorizes mechanism discovery from deployment policy.

Datasets

The authors evaluated AROMA across several benchmarks:

  • Proc-Count-Causal (v1-v4): A new procedural counting dataset designed for controlled mechanistic discovery and confirmation.
  • TallyQA: A standard benchmark for counting in natural, real-world images.
  • HoloCount: A holistic visual counting benchmark used to test the boundaries of the transferred policy.
  • GQA: Used as an "off-target" stress test to ensure that steering counting behavior doesn't accidentally break unrelated reasoning tasks.

Key Insights and Ideas

  • The Sensitivity-Energy Mismatch: One of the most striking findings is that the top four dimensions explaining ~90% of a head's sensitivity to counting only account for ~9.5% of its activation energy. This explains why standard "activation steering" often feels under-powered; you are pushing on a lever that the model barely "notices" unless you normalize the magnitude.
Figure 2. Sensitivity is not activation energy. (a) The top four dimensions explain 89.82% of the uncentered cardinality- sensitivity second moment and 86.78% after centering; both are far above the rank-matched null distribution. (b) The same Top-4 subspace contains little of the pooled activation norm: median ρ = 0.309, corresponding to only 9.52% median projected activation energy. This mismatch predicts why same-α low-rank scaling under-actuates and motivates magnitude normalization.
  • Control is a Stack, Not a Vector: The paper argues that "finding a causal component" is not the same as "controlling it." Reliable deployment requires a policy that manages target selection and gating as much as the geometry of the edit itself.
  • Iterative Re-linearization: Because the model's internal "decision boundaries" rotate as you apply an intervention, a single-shot steering vector is insufficient. MEA's iterative approach is crucial for "reachability"—actually getting the model to output the desired number.

Key Results

  • Reachability: MEA reached the correct oracle target in 81.25% of cases, compared to just 15.18% for a fixed-direction baseline under the same budget.
  • Accuracy Gains: On untouched procedural data, AROMA improved exact-count accuracy by +8.00 percentage points (pp).
  • Natural Image Success: On a cohort of natural images from TallyQA, the runner-up MEA policy improved the baseline by +4.286 pp, successfully repairing 51 errors while breaking only 4 previously correct ones.
  • Efficiency: The system achieved these gains while intervening on only 0.788% of the total population, demonstrating the power of selective gating.
Figure 3. Iterative MEA significantly outperforms frozen-direction and activation-aligned controls.

Limitations and Future Work

  • Task Specificity: Currently, the full MEA pipeline is only demonstrated for cardinality (counting). Whether this specific "iterative re-linearization" generalizes to more abstract reasoning tasks remains to be seen.
  • Model Specificity: While the hierarchy of control (low-rank geometry, energy mismatch) replicated across Llama and IDEFICS, the specific gains and optimal hyperparameters are model-dependent.
  • Target Selection: The "runner-up" rule is simple and effective but can fail if the correct answer isn't the second-highest probability. Future work could involve more sophisticated target proposal mechanisms.
  • Domain Shift: The gating policy required recalibration when moving from synthetic to natural images, suggesting that "when to act" is highly sensitive to data distribution.

Summary updated

Abstract

Mechanistic localization does not by itself yield usable control: causal sensitivity may occupy little activation energy, an accurate local direction may fail at finite scale, and a reachable numeral target may be wrong. We introduce AROMA (Adaptive Representation-Oriented Multimodal Actuation), a frozen-VLM control framework separating where to intervene, which geometry and gain to use, how to reach a target, which target to pursue, and when to act. In Llama-3.2-11B-Vision-Instruct, four dimensions of a causally validated cross-attention head explain 89.82% of cardinality sensitivity but only 9.52% of median activation energy; magnitude-normalized causal subspace actuation converts this mismatch into finite control, improving exact-count accuracy on untouched procedural data by +8.00 pp. Iterative Minimal-Effort Actuation (MEA) reaches 81.25% of oracle targets versus 15.18% for a frozen direction under matched step budgets. With target and gate frozen before evaluation, targeted MEA improves the previous frozen policy from 34.444% to 38.730% on a new natural Stage-2 cohort (N=630; paired p=4.63e-7) and yields 65.741% versus 65.153% over all 7,992 examples while editing only 63 inputs. Prospective routing, replication across heads and models, and comparisons with attribution-steering baselines further characterize each stage, separating localization, actuation, reachability, target quality, and selective deployment as distinct control problems.
BibTeX
@misc{zhang2026aroma,
  author = {Zhaoning Zhang},
  title = {{AROMA: Adaptive Representation-Oriented Multimodal Actuation for Vision-Language Model Control}},
  year = {2026},
  eprint = {2611.15041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15041},
}
Figure