Some vision-language models mix the text prompt into the image from the very first layer. This design is called early fusion, and it means the model does not just read out an answer at the end: the wording steers which visual features get computed in the first place. This proposal asks what happens to such models when the image is nearly dark. Our hypothesis is that as visual signal fades, the prompt's influence on the visual features grows. Two phrasings that mean the same thing ("the red mug on the left" versus "the left-hand red cup") then produce measurably different internal image features and different answers. If so, prompt sensitivity is not a fixed property of a model. It rises as the scene gets darker, and the rate of that rise depends on how early text and image are mixed.
The work has two parts. The first is a measurement study. We darken standard benchmarks with a physically realistic camera-noise pipeline, vary the prompt wording, and vary how early fusion begins, then trace where in the network paraphrases first diverge. The second is a targeted fix whose value depends on what the measurement finds. If the divergence is born in the early layers, we train the early visual features so they no longer depend on how the prompt is phrased under darkness. If the divergence only appears near the output, the cheap fixes are enough and we will say so.
Core Claim
Two findings motivate this, and neither has been connected to the other. DarkEQA reverses sRGB images into linear sensor space and adds shot, read, row-pattern and quantization noise over five darkness levels. It shows InternVL3.5-8B falling from 72.77% to 36.77% accuracy, and some models dropping below a text-only baseline at the darkest level. But its questions use fixed templates, so it cannot tell us whether wording matters more in the dark. MERLIM shows that instruction-tuned vision-language models swing by up to 26.5% across equivalent phrasings and often answer from language priors rather than pixels. It does this only on well-lit images. Putting the two together suggests a specific mechanism. When image tokens carry little information, cross-attention in an early-fusion encoder is dominated by text tokens, so surface features of the wording leak into the image representation before any reasoning happens.
The claim is falsifiable in three ways. First, the spread of accuracy across paraphrases should grow as darkness increases. Second, that growth should be steeper when fusion starts earlier. Third, the layer at which paraphrase-induced feature differences first become large should move earlier as darkness increases. If none of these holds, the proposal stops at the measurement paper, which is still new: prompt sensitivity has never been measured as a function of physical signal-to-noise ratio.
Research Program
The main testbed is EVF-SAM. It feeds the text prompt and image patches jointly into a BEIT-3 encoder from the first layer, then passes the result to the Segment Anything mask decoder. It reports an average cIoU of 79.0 on RefCOCO/+/g, with 1.32B parameters. We chose it because early fusion is its defining feature and because referring expressions have natural paraphrases. The paper evaluated neither darkness nor rephrasing. Its early fusion also gives us a clean experimental control, which we call fusion depth. We mask text-to-image attention in the first $k$ encoder layers, so image patches are encoded alone until layer $k$. Sweeping $k$ from 0 (EVF-SAM as published) to the full depth (late fusion) gives a family of models that differ only in when mixing begins. We fine-tune each masked variant briefly on clean RefCOCO so that the comparison is not confounded by removing attention the model was trained with.
We build the darkened data by applying the DarkEQA degradation ladder (levels L0 to L5) to RefCOCO, RefCOCO+ and RefCOCOg images. For each referring expression we create about eight paraphrases. These combine rule-based rewrites (synonym swaps, reordering of modifiers, relative-clause versus adjective forms) with the five syntactic template variations used in MERLIM. A language model proposes further rewrites, and humans check that each still picks out the same object. The main measurements at each darkness level and fusion depth are mean cIoU across paraphrases, worst-paraphrase cIoU, the max-minus-min spread, and the average mask IoU between predictions for paraphrase pairs. The last one captures disagreement even when both answers are equally wrong.
For the mechanistic analysis we compute, at every encoder layer, the distance between image-token features produced under two paraphrases of the same expression on the same darkened image. We normalize it by the distance produced by two expressions that refer to different objects. This ratio tells us at which depth the wording starts to change what the model "sees," and whether that depth moves earlier as light drops.
The intervention is used only if the analysis finds early-layer divergence. We fine-tune the encoder so that its early image features on a darkened image, prompted with any paraphrase, match the features a frozen copy of the model produces on the well-lit image with the original expression:
$$\mathcal{L}_{\text{inv}} = \sum_{l \le k^*} \big\| f_l(\tilde{x}, p') - \mathrm{sg}\, f_l(x, p) \big\|_2^2 .$$
Here $f_l(\cdot,\cdot)$ is the image-token feature map at layer $l$, $x$ is the clean image, $\tilde{x}$ its darkened version, $p$ the original expression, $p'$ a sampled paraphrase, $\mathrm{sg}$ a stop-gradient on the frozen teacher, and $k^*$ the depth found by the analysis. We add this loss to the usual segmentation loss.
The clean-image teacher borrows the dark-to-bright alignment idea of Watching it in Dark, which aligns dark and bright feature distributions for detection and recognition. Our version differs in two ways. The target is a prompt-conditioned feature, and the invariance also covers wording. This matters because the ExDark study shows ordinary features split dark and bright images of the same class into disjoint clusters (Getting to know low-light images). Aligning features for one fixed prompt could still leave them wording-dependent.
The negative-correlation result in Provable Dynamic Fusion places a constraint on the design. That paper proves fusion helps only when a modality's weight falls as its error rises. So we also log whether text-to-image attention mass in early layers rises as images darken. That would be the wrong direction for image features, since text would be reshaping them precisely when they are least reliable. We use this log as a diagnostic, not as a new gating module.
To check that the effect is not an artifact of the one architecture, we run the same paraphrase-by-darkness grid on DarkEQA with LLaVA-1.6-7B and InternVL3.5-8B. These are late-fusion models. The prediction is a flatter rise in sensitivity than for early-fusion EVF-SAM.
Baselines
The naive reference point is EVF-SAM as released, evaluated on darkened images with the original expressions only. Nobody has reported this. There is no reported result found on Lacuna for EVF-SAM under low light or paraphrase, and the only anchor is its clean 79.0 average cIoU. Every other comparison is measured against how far this reference falls.
The strongest competitor for the sensitivity question is paraphrase ensembling at test time. Run all paraphrases and fuse the masks by pixel-wise voting, with an optional per-class weighting in the spirit of CARPRT. CARPRT reweights prompt templates using unlabeled test images and reaches 67.45% against 64.26% for mean ensembling on ten clean classification datasets. It is the strongest competitor because it removes wording variance by design and needs no training. Its cost is the fairness point. In early fusion, every paraphrase requires a full re-encoding of the image, so an ensemble of $K$ phrasings costs $K$ times the compute. We report ensembling at $K$ = 1, 4 and 8 alongside inference time, and our fine-tuned single-pass model must match the $K$ = 8 ensemble's worst-paraphrase cIoU at L3 to L5 to count as a win. We also note CARPRT's own warning: when base accuracy is low, its pseudo-labels reinforce errors. Darkness is exactly that regime, so we expect the weighted variant to degrade faster than plain voting, and we will check.
The strongest competitor for the low-light question is enhancement before segmentation. We run DarkIR, the enhancer DarkEQA used, and Zero-DCE on the darkened images, then feed the results to the unmodified model. DarkEQA's figures set expectations. With DarkIR, LLaVA-1.6-7B rose from 39.25% to 54.03% at L5, but performance fell at L1. A win for our method means beating enhancement at every darkness level on worst-paraphrase cIoU, not only at L5.
The essential ablation is darkness-augmented fine-tuning with the segmentation loss alone, using the same data, steps and paraphrase sampling but without $\mathcal{L}_{\text{inv}}$. A second ablation computes the invariance only on final output masks rather than on early features. A third uses a teacher that sees the same paraphrase $p'$ on the clean image, which keeps darkness invariance but drops wording invariance. All fine-tuned variants get identical compute and the same hyperparameter search budget, tuned on a held-out split of paraphrases and images that is never used for reported numbers.
Why not just…?
Why not just brighten the image first? Enhancement addresses darkness but not wording. On a fixed prompt it may close part of the accuracy gap, but nothing in it stops two paraphrases from steering an early-fusion encoder differently on the residual noise. Enhancers also add artifacts. DarkEQA found DarkIR hurt accuracy at L1, and both the CLIP-prior enhancement work and YOLA report that images tuned for human viewing lose detail that downstream models need. The paraphrase-spread measurement on enhanced inputs settles this directly. If enhancement flattens the sensitivity-versus-darkness curve, the mechanism we propose is not the bottleneck.
Why not just ensemble paraphrases at inference? This is the real threat, and we treat it as the main baseline rather than arguing it away. On clean data, prompt ensembles reliably help. Our case rests on cost and on the dark regime. Early fusion makes each extra phrasing a full forward pass, and averaging many wrong masks does not produce a right one when every paraphrase is misled. If an 8-way vote matches our single-pass model at L3 to L5, we will report that the fix is ensembling and that feature training buys only speed.
Why not just fine-tune on darkened images, or learn continuous prompts? Darkness-augmented fine-tuning is exactly our first ablation. It may by itself make features less wording-dependent, and if it matches $\mathcal{L}_{\text{inv}}$ the specific ingredient adds nothing. Learned continuous prompts in the style of CoOp do not fit this setting. They replace a fixed class template, and a referring expression is a free-form query written by the user, so there is no single template to learn. We include a CoOp-style learned prefix prepended to every expression as a secondary baseline.
Why not just use late fusion, if early fusion is the fragile part? It may turn out that it is. The depth sweep measures exactly this trade-off. EVF-SAM reports that early fusion beats much larger late-fusion LLM pipelines on clean referring segmentation, and the question is whether that advantage survives darkness. If the curves cross, and late fusion wins beyond some darkness level, that crossing point is itself a useful design rule.
Decisive Checks
The first check is the sign of one slope. Paraphrase spread must increase from L0 to L5 for EVF-SAM, and more steeply for $k$ = 0 than for masked-fusion variants. We will confirm this on real darkness by prompting with class names in varied templates on ExDark and scoring the predicted mask's bounding box against ExDark's boxes, so the result is not specific to synthetic noise. The second check is that the layer-wise divergence ratio crosses its threshold earlier as darkness increases. If it does not, the early-layer loss is unjustified and we drop it. The third check is the head-to-head at L3 to L5: $\mathcal{L}_{\text{inv}}$ must beat darkness-augmented fine-tuning on worst-paraphrase cIoU by a margin larger than seed-to-seed variation over three seeds, match the 8-way paraphrase ensemble at one-eighth the inference cost, and keep clean L0 cIoU within one point of 79.0. The DarkEQA paraphrase grid on late-fusion models serves as an outside test of the mechanism. A flatter slope there supports the claim that fusion depth, and not darkness alone, drives the growth in prompt sensitivity.
References
- DarkEQA: Benchmarking Vision-Language Models for Embodied Question Answering in Low-Light Indoor Environments
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- EVF-SAM: Early Vision-Language Fusion for text-prompted Segment Anything Model
- Watching it in Dark: A Target-Aware Representation Learning Framework for High-Level Vision Tasks in Low Illumination
- Getting to know low-light images with the Exclusively Dark dataset
- Provable Dynamic Fusion for Low-Quality Multimodal Data
- CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Vision-Language Model
- Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement
- Unsupervised Image Prior via Prompt Learning and CLIP Semantic Guidance for Low-Light Image Enhancement
- You Only Look Around: Learning Illumination-Invariant Feature for Low-light Object Detection
- Learning to Prompt for Vision-Language Models