Where Does Arousal Go? Measuring What Survives the Hand-off Between Vision-Language Agents

A common way to make large models more reliable is to split a task across several model instances, each with a narrow role: one describes, one reasons, one checks. This works well for math and code, where the information passed between agents is already symbolic. It is much less clear that it works for perception. When the first agent turns an image into words, anything that resists words is lost before the later agents ever see it. We propose to measure this loss directly. The measuring instrument is a set of standardized emotional picture databases: the International Affective Picture System (IAPS), the Geneva Affective PicturE Database (GAPED), and the Open Affective Standardized Image Set (OASIS). Large human panels have rated every image in these sets for pleasantness (valence), intensity (arousal), and, in IAPS, sense of control (dominance). These norms provide a continuous, externally fixed ground truth that can be read out of any intermediate message in a multi-agent pipeline.

The claim is simple. In multi-agent vision-language pipelines, the text hand-off is the main bottleneck for perceptual quantities. Arousal is the first quantity to be lost. Decomposing the task helps only when downstream agents keep access to perceptual evidence rather than only the upstream agent's prose. We will build a probe that scores how much of the human-rated affect survives each message. We will use it to compare text-only decomposition against "evidence-carrying" decomposition, in which later agents also receive the image and a calibrated numeric measurement. We will also report honestly whether either beats a plain linear readout from image embeddings.

Why this question, and why now

Two findings in the literature point the same way, and nobody has connected them. Alrasheed et al. asked GPT-4 Turbo to rate the 730 GAPED images zero-shot. Correlation with human norms was r = 0.87 for valence but only r = 0.72 for arousal. On 3-point Likert bins, accuracy was 77% for valence and 57% for arousal. When the model rated from its own textual description instead of the image, both dropped, to 0.79 and 0.65. Conwell and Hamblin found something similar for aesthetic judgments on OASIS. Linear readouts from vision-language embeddings explained 80.5–87% of the explainable variance, while machine-generated captions explained about 39%. Both results say that verbalizing an affective image throws information away.

Meanwhile the multi-agent literature reports large gains from separating perception or planning from execution and verification. MACM's thinker–judge–executor split raised GPT-4 Turbo on Level 5 MATH from 54.68% to 76.73% (Lei et al.). Similar splits reduced hallucination in geometry (Mouselinos et al.) and design-rule code (Ho). The same MACM paper, however, notes that these systems do markedly worse on problems needing visual or spatial imagery, and the survey by Li et al. flags cascades in which a downstream agent treats an upstream error as fact. Our proposal treats the caption-loss findings as the explanation for that perceptual weakness, and affective norms as the cleanest place to quantify it.

The closest existing system is EmoAgent (Mao et al.). It uses planner, editor, and critic agents to edit images toward a target emotion category, reaching 64.58% emotion accuracy on EmoEditSet versus 54.46% for EmoEdit. EmoAgent is generative, works with categorical labels, and never asks what information its agents exchange. The appraisal agent of Hong, Sun, and Chen decomposes emotional reasoning into appraisal stages, but only over dialogue text. Zhang et al. find an internal split inside one vision-language model: middle-layer attention heads gather emotional visual cues, and deeper layers write the narrative. That mechanistic split motivates our external decomposition, but they study it inside a single forward pass and on categorical benchmarks.

Core Claim

We make three testable claims. First, affect that is linearly decodable from an image embedding is substantially less decodable from the text messages a perceiver agent sends downstream. The loss is larger for arousal than for valence, consistent with Alrasheed's description-only drop. Second, a text-only decomposition, in which a perceiver describes, an appraiser rates, and a critic checks the text, will not reliably beat a single prompt on arousal, because the appraiser and critic inherit the perceiver's losses. Third, giving downstream agents the image and a calibrated numeric affect measurement recovers the lost signal. It also lets the critic catch polarity errors, where a distressing scene is rated as pleasant. Whether that recovery beats a plain embedding probe is the question we treat as open.

Research Program

The central instrument is a retention score for any message $m$ passed between agents. We embed the message with a fixed sentence encoder and fit a cross-validated ridge regression from that embedding to the human normative rating on dimension $d$ (valence, arousal, or dominance), giving held-out variance explained $R^2_d(m)$. We do the same from a frozen vision-language image embedding of the stimulus, giving $R^2_d(\text{img})$. Retention is the ratio

$$\rho_d(m) = \frac{R^2_d(m)}{R^2_d(\text{img})},$$

which equals 1 when the message carries as much linearly readable affect as the image itself. Because the probe is linear and fixed across conditions, differences in $\rho$ reflect what the message contains, not how clever the reader is. We will also report the noise ceiling, from split-half reliability, wherever rater-level data permit. This lets losses be read against what humans themselves agree on.

We will build three pipeline variants on the same vision-language model, so that only the information flow changes. In the text-only variant, a perceiver agent writes a neutral description of the scene. Following Zhang et al.'s split between gathering cues and writing the narrative, it is told to list objects, people, and physical cues without emotional language. An appraiser agent sees only that description and outputs valence, arousal, and dominance on the dataset's scale. A critic sees the description and the ratings and may send one round of feedback, in the style of MACM's judge. In the evidence-carrying variant, the roles are the same, but the appraiser and critic also receive the image. They also receive a measurement from a tool: a ridge probe on image embeddings, trained on a different stimulus set so that it never sees the test images. The critic is instructed to flag disagreements larger than a set threshold between the appraiser's rating and the tool's reading. The third variant is the monolithic single prompt, reproducing the Alrasheed protocol.

We log every inter-agent message and compute $\rho$ at each hop. This gives a map of where valence and arousal leave the pipeline, and whether the critic ever puts them back.

We will run one open-weight vision-language model, which also lets us extract embeddings, and one closed API model of the GPT-4 Turbo class, so that the monolithic condition is directly comparable to Alrasheed's numbers. The stimulus sets are GAPED (valence and arousal on 0–100 scales), OASIS (900 images, valence and arousal), and IAPS (9-point SAM ratings, the only one with dominance). The probe tool is trained on OASIS and tested on GAPED and IAPS, and the reverse. This cross-set setup also exposes scale and domain shift.

As a secondary analysis, if we can obtain the per-image norms behind Huang et al.'s Chinese and American IAPS ratings, we will ask a narrower question. Does the culture-specific part of the rating, meaning the difference between the two cohorts' norms, survive in perceiver messages at all? That difference concentrates on social and erotic scenes, exactly the content a perceiver is likely to describe blandly. This is a stress test of retention, not a cultural-panel system.

Baselines

The naive reference point is predicting the dataset mean for every image, which fixes r = 0. Alongside it we include the monolithic zero-shot prompt run exactly as in Alrasheed et al.. On GAPED that protocol reported Pearson r = 0.87 for valence and r = 0.72 for arousal, with Likert accuracy of 77% and 57%. Its description-only variant reported 0.79 and 0.65. We rerun this baseline ourselves on both models rather than relying only on the published figures. Beating it means an arousal correlation clearly above our own rerun, with bootstrap confidence intervals over images that do not overlap. The published 0.72 is a sanity check that our rerun lands in the same range.

Because multi-agent pipelines spend more tokens, we add a compute-matched monolithic baseline. It samples the single prompt as many times as the pipeline makes calls, and averages the ratings. Any pipeline gain smaller than this averaging gain is attributed to extra compute, not decomposition.

The strongest competitor is a ridge regression on frozen vision-language image embeddings, the method of Conwell and Hamblin. They report 80.5–87% of explainable variance for aesthetic ratings on OASIS. That is an aesthetic, not a valence or arousal, result; we found no reported result on Lacuna for a CLIP-style probe on valence or arousal for GAPED or IAPS, so we will establish it. The probe is trained with the same labeled images the pipeline's tool sees and tuned by nested cross-validation. The pipeline gets an equal budget of prompt-tuning trials on a held-out development split.

We also include the ResNet50-plus-XGBoost model of Arslan et al., reported at 98% accuracy (AUC 0.981) on 3-class GAPED sentiment. That task largely mirrors GAPED's category structure, so we retrain it as a regressor on continuous ratings and report it under the same splits.

The key ablation isolates evidence access. It is the evidence-carrying pipeline with the image withheld from downstream agents but the tool reading kept, and then with the tool removed but the image kept. This separates whether gains come from seeing the pixels again or from a calibrated number.

Why not just…?

Why not just use a single, better prompt? Alrasheed et al. already tried few-shot prompting on GAPED, and it did not reliably improve, and sometimes hurt, accuracy. Their 57% arousal Likert accuracy is the gap at issue. A compute-matched self-consistency baseline is included precisely so that a pipeline cannot win simply by spending more. If averaged monolithic samples match the evidence-carrying pipeline, we will report that decomposition adds nothing beyond sampling.

Why not just train a linear probe on image embeddings and skip agents entirely? This is a real threat. On mean-rating prediction with ample labels, we expect it to win or tie, given Conwell and Hamblin's caption-versus-embedding gap. We do not argue around this. We run the comparison across label budgets of zero, 50, 200, and all available images. The probe cannot run at zero labels, while the pipelines can. We report the crossover point. We also report polarity-flip rates on negative images, where the critic has a chance to add something a probe cannot. If the probe dominates at every budget, the contribution is the retention diagnostic and a negative result for decomposition on perceptual rating, which is itself useful to the multi-agent literature.

Why not just steer the model internally, as VEENA does? Zhang et al. report a 6.7% average gain on MER-UniBench from amplifying emotion-related attention and neurons. However, that gain is on categorical recognition, it needs white-box access, and it does not tell us anything about agent hand-offs. Where our open-weight model permits, we will apply this kind of steering to the monolithic condition as an additional point of comparison. The retention analysis is orthogonal to it.

Proposed Experiments

The first decisive check is the retention map. For text-only pipelines on GAPED, OASIS, and IAPS, we expect $\rho_{\text{arousal}}$ at the perceiver message to fall well below $\rho_{\text{valence}}$, mirroring the description-only drop Alrasheed observed. If perceiver messages retain arousal as well as valence, the bottleneck hypothesis fails, and any arousal deficit must come from the appraisal step instead.

The second check is the three-way comparison of monolithic, text-only, and evidence-carrying pipelines. We measure Pearson r against norms, Likert accuracy (for comparability with Alrasheed), and the rate at which images with normatively negative valence receive positive ratings. We predict that text-only does not beat compute-matched monolithic on arousal, while evidence-carrying does. The ablation then attributes the gain to the image, the tool, or both. The probe comparison across label budgets settles whether agents earn their cost anywhere.

Finally, we ask whether the critic ever increases retention. That is, does any critic message carry more readable arousal than the perceiver message it responds to? This tests whether verification in this setting can recover lost perceptual information or only filter what it is given. A "no" would turn the cascade concern in Li et al. into a measured property of perceptual pipelines.