Abstract
Pretrained visual-language models have made significant advancements in multimodal tasks, including image-text retrieval. However, a major challenge in image-text matching lies in language bias, where models predominantly rely on language priors and neglect to adequately consider the visual content. We thus present Multimodal ASsociation Score (MASS), a framework that reduces the reliance on language priors for better visual accuracy in image-text matching problems. It can be seamlessly incorporated into existing visual-language models without necessitating additional training. Our experiments have shown that MASS effectively lessens language bias without losing an understanding of linguistic compositionality. Overall, MASS offers a promising solution for enhancing image-text matching performance in visual-language models.
Introduction
Vision-Language Models (VLMs) such as CLIP, OFA, and BLIP-2 have transformed the landscape of multimodal information retrieval. By learning joint representations of images and text from massive datasets, these models enable sophisticated search and captioning capabilities. However, a critical flaw persists: language bias. Because these models are trained on vast text corpora, they often develop internal linguistic priors that overshadow visual evidence. When a model predicts a label or matches a caption, it may be "guessing" based on what sounds likely in English rather than what is actually present in the pixels.
The Challenge: Language Priors vs. Visual Evidence
Language bias manifests as a form of multimodal hallucination. For instance, if a model encounters the text prefix "A firefighter carries a...", its internal language component might assign a high probability to the word "man" simply because that pairing is statistically frequent in its training data. This occurs even if the accompanying image clearly depicts a woman.
This overreliance on text priors leads to three primary failure modes:
- Social Bias: The reinforcement of gender or racial stereotypes in retrieval results.
- Poor Compositionality: An inability to distinguish between sentences with identical words but different meanings (e.g., "the cat chases the dog" vs. "the dog chases the cat").
- Visual Neglect: Ignoring specific visual details, such as the exact number of objects, in favor of common linguistic patterns.
The MASS Framework
To address these issues, we introduce the Multimodal ASsociation Score (MASS). MASS is a training-free, inference-time framework designed to isolate the true association between an image and a text string by filtering out the model's internal linguistic noise.
Pointwise Mutual Information (PMI)
The core of MASS is rooted in Information Theory, specifically Pointwise Mutual Information (PMI). Instead of relying solely on the conditional likelihood of a text given an image ($p(\text{text}|\text{image})$), MASS calculates how much the presence of the image increases our confidence in that text compared to the text's baseline probability ($p(\text{text})$).
The debiased score is defined as: $$\text{Score}_{\text{MASS}} = \log p(\text{text} | \text{image}) - \log p(\text{text})$$
By subtracting the linguistic log-likelihood (the second term), the model effectively removes the "prior" influence of common phrases, forcing the scoring mechanism to focus on tokens that are specifically supported by the visual modality.
The "Null Image" Innovation
A significant challenge in implementing PMI is that calculating the true marginal probability $p(\text{text})$ is computationally expensive, typically requiring an average across thousands of random images. MASS introduces a clever shortcut: using a "null image" (a black-filled image) as a proxy for the absence of visual information. This allows MASS to be computed with just two forward passes of an off-the-shelf model, making it a "plug-and-play" solution that requires zero additional training or fine-tuning.
Experimental Evaluation
We evaluated MASS across several benchmarks designed to expose different facets of language bias:
- Natural Color Dataset (NCD): Tests whether models can identify "gray" fruits when linguistic priors strongly expect colors like "red" or "yellow."
- VALSE: A benchmark for spatial relationships and counting, particularly the "adversarial" subset where object counts are unusual or counter-intuitive.
- MS-COCO (Gender Bias): Evaluates retrieval fairness using gender-neutralized captions (e.g., using "person" to retrieve images of both men and women).
- Winoground & SVO-Probes: High-difficulty datasets that test subject-verb-object relationships and linguistic compositionality.
Results and Discussion
Our experiments demonstrate that MASS consistently improves performance across diverse architectures, including OFA, BLIP-2, and LLaVA.
- Compositionality: On the notoriously difficult Winoground benchmark, MASS enabled the OFA-large model to achieve a Group score of 20.3%, making it the only method in our study to surpass random chance (16.7%).
- Counting Accuracy: On the VALSE adversarial set, MASS improved accuracy by 4.7%, indicating a shift from linguistic guessing to actual visual counting.
- Debiasing and Retrieval: In text-to-image retrieval on MS-COCO, MASS significantly reduced gender bias (Bias@1) while simultaneously increasing Recall@1 from 38.0 to 55.6. This suggests that removing bias does not just improve fairness—it makes the model a more accurate search engine.
Conclusion
MASS acts as a high-pass filter for multimodal information, ignoring the "obvious" linguistic patterns that a model could guess without looking and emphasizing the specific details supported by the pixels. While MASS is dependent on the underlying quality of the visual encoder, it provides a robust, computationally efficient method for ensuring that Vision-Language Models actually "look" at the image before they speak.
Summary updated