TL;DR
Vision-language models (VLMs) that use autoregressive captioning likelihood for image-text matching frequently misfire because they rely on textual priors rather than visual evidence—for example, predicting a tomato is "red" even when presented with a grayscale image. The authors propose MASS (Multimodal ASsociation Score), a training-free, inference-time framework that subtracts language-only priors using a simple "null" (black-filled) image. By isolating pure cross-modal mutual information, MASS significantly diminishes demographic and concept biases while sharply improving compositional reasoning.
What Problem Does This Paper Address?
Dual-encoder models trained with contrastive objectives (like CLIP) are notorious for treating sentences as "bags of words," failing to parse complex syntactic relationships or compositional details. Autoregressive captioning models (such as OFA, BLIP-2, and LLaVA) understand linguistic structure much better, and their token log-likelihoods can serve as image-text matching scores.
However, generative captioners introduce a major flaw: language bias. Because they are trained to produce fluent text, their likelihoods over-index on common textual sequences regardless of the visual input. A model might prefer "fireman carries hose" over "firewoman carries hose" or guess "red tomato" over "gray tomato" simply because the phrase appeared more frequently during pretraining. This paper aims to strip away these text-only priors at inference time without requiring retraining or fine-tuning.
Methods
MASS reformulates image-text scoring through the lens of Pointwise Mutual Information (PMI):
$$\text{PMI}(x; c) = \log \frac{p(x|c)}{p(x)}$$
By Bayes' rule, the image-conditioned log-likelihood decomposes into a text-only linguistic prior and an association term:
$$\log p_\theta(x|c) = \underbrace{\log p_\theta(x)}_{\text{linguistic}} + \underbrace{\log \frac{p_\theta(c|x)}{p_\theta(c)}}_{\text{association}}$$
To isolate the visual association, MASS subtracts the token-level marginal linguistic likelihood $\log p_\theta(x_t \mid x_{<t})$ from the conditional likelihood $\log p_\theta(x_t \mid x_{<t}, c)$.
Because standard VLMs cannot compute $p_\theta(x)$ directly without an image input, standard approaches would require averaging forward passes over hundreds of random images (Monte Carlo sampling)—a computationally prohibitive step. The authors introduce a remarkably lightweight workaround: a blank, black-filled null image ($c_\emptyset$).
The final MASS similarity score is computed by averaging the log-ratio over sequence length $l$:
$$S_{\text{MASS}}(c, x) \approx \frac{1}{l} \sum_{t=1}^l \log \frac{p_\theta(x_t \mid x_{<t}, c)}{p_\theta(x_t \mid x_{<t}, c_\emptyset)}$$
Datasets
The authors evaluated MASS across five distinct benchmark suites:
- Natural Color Dataset (NCD): Grayscale fruit images (723 images, 20 fruit types) tested against "The fruit is gray" vs. "The fruit is [natural color]".
- VALSE Counting Benchmark: Adversarial and foil image-caption pairs measuring object-counting sensitivity.
- MS-COCO (Karpathy 5k split): Evaluated for gender bias in text-to-image and image-to-text retrieval using gender-neutralized and original captions.
- Winoground: 400 visuo-linguistic challenge pairs with identical words arranged in different syntactic permutations.
- SVO-Probes: Probing sensitivity to subject, verb, and object alterations in captions.
Key Insights and Ideas
- Text-only marginals as a negative prompt: Conditioning on a black image acts as an effective control baseline, prompting the model to reveal its default text preferences. Subtracting this baseline penalizes commonly hallucinated words and rewards tokens that genuinely depend on visual context.
- Debiasing improves—rather than hurts—compositionality: Prior debiasing techniques (such as CLIP-clip) often degrade retrieval accuracy to achieve fairness. Because MASS targets the root statistical confounder (prior probability), it simultaneously reduces demographic bias and improves fine-grained linguistic comprehension.
- Universal plug-and-play compatibility: MASS requires zero parameter updates and works out of the box with diverse backbones, including encoder-decoder frameworks (OFA) and large visual instruction-tuned models (LLaVA, BLIP-2).
Key Results
- Color Association (NCD): Token Likelihood (TL) was biased in 33.33% of fruit/vegetable categories (preferring the natural color over the visible gray color). MASS reduced biased categories to 0%.
- Counting (VALSE): On the difficult adversarial split (small-number foils vs. large-number ground truths), OFA-large + MASS scored 76.7%, dramatically outperforming raw token likelihood (66.8%) and sequence-level matching (72.0%).
- Gender Bias & Retrieval (MS-COCO):
- In text-to-image retrieval, OFA-large + MASS maintained low gender bias while skyrocketing Recall@1 to 55.6% (compared to 38.0% for TL and 36.5% for CLIP).
- In image-to-text retrieval, MASS cut the Bias@1 metric nearly in half compared to TL (3.5 vs. 6.6) while achieving the highest Recall@1 (57.3%).
- Linguistic Compositionality (Winoground): OFA-large with MASS achieved a 20.3% Group Score (surpassing human-level chance of 16.7%), compared to 15.8% for TL and 9.5% for ITM. Similar improvements occurred when tested on BLIP-2 and LLaVA.
Limitations and Future Work
- Cannot rectify intrinsic representational gaps: Because MASS operates purely at inference time, it cannot extract visual features the underlying vision backbone failed to encode in the first place.
- Limited to language-driven biases: MASS addresses linguistic distribution skew, but it does not resolve visual representation biases (e.g., demographic disparities in the training image distribution) or complex multimodal intersectional biases.
- Heuristic null image: While a completely black image works well empirically across examined architectures, other visual backbones may respond idiosyncratically to out-of-distribution blank inputs, warranting further study into optimal marginal image proxies.
Summary updated