How Much of the Cultural Gap in Vision-Language Models Is Just Image Style?

Vision-language models such as CLIP, which learn to match images with captions, recognize a stove or a toothbrush far worse in a low-income household in Nigeria than in a high-income one in Germany. The cultural-representation literature documents this gap thoroughly. It usually prescribes one remedy: collect more diverse photographs. Medical imaging has faced a similar problem, where a model trained on CT scans fails on MRI, and it developed a different toolkit. That toolkit identifies the parts of an image that carry only sensor "style" (brightness, contrast, color, texture statistics), removes or randomizes them, and forces the model to rely on shape. We propose to bring that toolkit across, first as a measuring instrument and then as a targeted fix.

The core question is simple. When a frozen vision-language model fails on a low-income or non-Western photo of an everyday object, how much of the failure goes away if we change only the image's low-frequency appearance statistics and leave its geometry untouched? What remains is failure caused by the object genuinely having a different form, such as a charcoal hearth instead of an induction cooktop. We expect this split to vary strongly by object class. We also expect it to predict which remedy works: appearance randomization with no new data for "style-gap" classes, and concept knowledge or real data for "form-gap" classes.

Core Claim

The cultural gap on object-recognition benchmarks is a mixture of two things. One part is an appearance shift that behaves like a sensor shift in medical imaging. The other part is a change in the object's form that no style-invariance method can fix. The two can be measured separately, per class, with controlled frequency-domain interventions on real images and no generative model. The new contribution is this attribution itself, the claim that it predicts per-class gains from different remedies, and a cheap, data-free appearance-randomization fine-tune evaluated only where the attribution says it should help.

This framing addresses a tension the prior work leaves unresolved. Cross-modality methods assume a clean split between invariant content and swappable style. SIFA pairs image translation with feature alignment for MR-to-CT cardiac segmentation. The causal-mechanisms work uses mask-conditioned diffusion to change modality style while holding anatomy fixed and reaches 86.20% Dice on CT-to-MRI. Anatomy really is invariant across scanners. The Dollar Street paper, however, notes that a stove's function is universal while its form is not. Nobody has measured where the style assumption holds for cultural shift and where it breaks. Our experiments do exactly that.

Research Program

The instrument borrows from Fourier-domain work in medical imaging. FAN-Net normalizes only the low-frequency Fourier amplitude of stroke MRIs, with an optimal low-frequency band at alpha = 0.1, and leaves the phase alone. FAMNet shows that mid-frequency bands are the most similar across modalities. Two terms need a plain definition. The amplitude spectrum records how much of each spatial frequency an image contains, which roughly captures lighting, color cast, and coarse texture. The phase spectrum records where edges and contours sit.

For an image from a low-income household, we replace its low-frequency amplitude with the average low-frequency amplitude of high-income images of the same class and keep its phase. We then ask whether the frozen model's zero-shot prediction improves. We apply this intervention as a nested ladder of increasing strength. The mildest rung is global color and brightness histogram matching, which captures camera and lighting differences. The next is low-frequency amplitude replacement across several band widths. The strongest is full amplitude replacement, which begins to erode texture that may itself be semantic, such as mud versus steel. The ladder lets us report how much of the gap is acquisition-level versus material-level appearance. We do not assume frequency cleanly separates culture from style.

We define a style-attributable fraction for each class $c$:

$$\rho_c = \frac{A_c(\text{low}\to\text{high swap}) - A_c(\text{low}\to\text{low swap})}{A_c(\text{high}) - A_c(\text{low})},$$

The terms are as follows. $A_c(\text{high})$ and $A_c(\text{low})$ are the model's accuracies on unmodified high- and low-income images of class $c$. $A_c(\text{low}\to\text{high swap})$ is its accuracy on low-income images given high-income amplitude statistics. $A_c(\text{low}\to\text{low swap})$ is its accuracy on low-income images given amplitude statistics from other low-income images. That last term is a control: the swap operation itself introduces artifacts, and the control ensures they are not counted as recovered gap.

We run the reverse intervention as well, pushing high-income images toward low-income amplitude statistics. This is a second causal check. If style drives the gap, high-income accuracy should fall by a comparable amount. The same attribution is computed across GeoDE regions, with Africa and Europe as the anchor pair.

The remedies follow from the attribution. The style remedy borrows from SynthSeg. SynthSeg trains segmenters on deliberately unrealistic random contrasts and finds they generalize better than models trained on realistic variation. We lightly fine-tune the image encoder of an open CLIP-family model on Western-heavy ImageNet-style data only. During fine-tuning, the low-frequency amplitude is replaced with random, often implausible spectra and the phase is kept fixed. A distillation term holds the text-aligned embedding near the original model's so that zero-shot recognition is preserved. No target-culture images are used.

The form remedy borrows the anchoring idea from K-MaT, which regularizes learned prompts toward LLM-written clinical descriptions so they do not drift into modality shortcuts. Here we replace each class's single text prompt with a set of descriptions of that object's regional forms (for example, "a cooking stove, which may be a clay hearth, a kerosene burner, or a gas range"). This remedy uses no images. We deliberately do not port K-MaT's optimal-transport alignment of prompt geometry. It needs source and target prompt sets to align, and the attribution question does not depend on it.

Baselines

The naive reference point is the frozen zero-shot model with standard class-name prompts, evaluated on Dollar Street and GeoDE by income quartile and by region. In the file, no reported result was found on Lacuna for zero-shot OpenCLIP or SigLIP accuracy by income quartile on Dollar Street. We will measure it. For scale, No Filter reports an income-quartile gap of 32.5% for English-filtered SigLIP. Older ImageNet models score 18.0% versus 57.4% top-5 accuracy on the lowest versus highest quartile (Dollar Street).

The strongest data-free competitor at the pretraining level is multilingual, unfiltered pretraining. No Filter's globe-tl SigLIP improves Dollar Street by +1.44%, GLDv2 by +5.62%, and MaRVL by +1.66% over the English-filtered model. This comparison matters because it represents the field's current zero-new-annotation answer. It also costs web-scale retraining, whereas our fine-tune does not. If those checkpoints are available, we evaluate them directly. If not, we treat +1.44% on Dollar Street as the reference gain to exceed. Beating it means closing more of the low-versus-high quartile gap on Dollar Street than +1.44 points of overall accuracy while not losing ImageNet accuracy.

The strongest competitor overall uses real diverse data. GeoDE reports that adding its commissioned photos to ImageNet training gives a 9% improvement on Dollar Street and a 21% improvement on the GeoDE test set. Fine-tuning a ResNet on Dollar Street itself reaches roughly 75% across all income levels. We do not expect a data-free method to match these numbers. Instead, we fine-tune the same backbone on GeoDE with the same step count and learning-rate search as our amplitude-randomization run. We then ask what fraction of GeoDE's Dollar Street gain our method recovers on classes with high $\rho_c$ versus low $\rho_c$.

Two ablations isolate the key ingredient. The first is a matched generic-augmentation fine-tune using color jitter and blur with the same compute. It tests whether gains come specifically from phase-preserving amplitude randomization or from any photometric augmentation. The second is a realistic-range amplitude randomization, which samples only spectra observed in real images. It tests the SynthSeg finding that unrealistic variation generalizes better. All fine-tunes share data, steps, and a fixed hyperparameter budget of equal size.

Why not just…?

Why not just fine-tune on GeoDE or Dollar Street? It works. Its reported gains of 9% and 21% are the ceiling we compare against, not an alternative we dismiss. The limitation is cost, about $0.87 per image and $54,000 for GeoDE, and the fact that it tells you nothing about where the next dollar should go. The attribution answers that question. If most classes turn out to have low $\rho_c$, the honest conclusion is that data collection is unavoidable and appearance invariance is a dead end for this problem, and we will report that.

Why not just use standard color augmentation? This is the most serious threat. If the generic photometric-augmentation fine-tune matches amplitude randomization, then the medical-imaging machinery adds nothing beyond cheap augmentation. The matched ablation above settles it. The attribution study still stands either way, because it uses interventions to measure the gap rather than as a training method.

Why not just scale up or translate the prompts? CVLUE found that translating text while keeping Chinese imagery leaves performance poor. It also found that larger zero-shot models are beaten by smaller locally fine-tuned ones. CULTURALVQA reports GPT-4o falling from about 72% on US questions to about 43% on Nigerian ones. We will still run the attribution across model sizes. If $\rho_c$ shrinks as models grow, scale is absorbing the style part of the gap, which is itself informative.

Why not just restyle images with a diffusion model? Structure-conditioned generation is the method of the causal-mechanisms work. That same paper reports it can blur fine boundaries. Beyond Aesthetics and Deconstructing Bias show that text-to-image models render non-Western subjects as caricatures and break when composing non-Western cultural elements. A diffusion intervention would inject exactly the stereotyped content we are trying to measure. Fourier swaps add no new content, which is why we use them as the instrument.

Proposed Experiments

The first experiment computes $\rho_c$ for every Dollar Street topic and every GeoDE class, across the intervention ladder and for two or three open model families. We report bootstrap confidence intervals over images and the reverse-swap check.

We validate the attribution against independent human judgment. Annotators blind to model outputs label each class by whether its physical form varies across income or region. Examples include the stove in Dollar Street and GeoDE's dustbin, where models score only 37.3% (GeoDE). The prediction is that form-variant classes have low $\rho_c$.

The decisive test is a crossover interaction. Amplitude-randomization fine-tuning should gain most on high-$\rho_c$ classes. Regional-description prompting should gain most on low-$\rho_c$ classes. The per-class correlation between $\rho_c$ and each remedy's gain should have opposite signs. If both remedies help the same classes, or $\rho_c$ predicts neither, the core claim fails.

A secondary transfer test uses CVLUE retrieval and grounding. There, Qwen-VL scores 46.9% IoU on categories shared with MS-COCO and 34.8% on Chinese-specific ones. We predict that the style intervention recovers more on the shared categories and little on the culture-specific ones. We also check that ImageNet zero-shot accuracy stays within one point of the original model, so that any gain is not bought by losing standard performance.