TL;DR
Current compositionality benchmarks for Generative Vision-Language Models (GVLMs) suffer from a severe syntactical bias, allowing models to "cheat" using their internal language model priors without actually grounding their answers in the image. To address this, the authors introduce the SyntaxBias Score to quantify this loophole and propose SADE, a new syntactically de-biased benchmark that forces models to rely on true visual understanding rather than grammatical fluency.
What Problem Does This Paper Address?
Older, encoder-based vision-language models (like CLIP) famously suffer from a "bag-of-words" problem—they struggle to tell the difference between "a dog bites a man" and "a man bites a dog" because they focus on individual tokens rather than word order and syntax.
Enter Generative Vision-Language Models (GVLMs) like LLaVA and MiniGPT-4. By fusing visual encoders with powerful Large Language Models (LLMs), these models inherently possess a deep understanding of grammar. To evaluate them, researchers typically use VisualGPTScore, which measures the log-likelihood of the model generating a specific caption given an image.
However, this paper uncovers a major flaw: existing compositionality benchmarks create "hard negative" captions by swapping or shuffling words, resulting in grammatically awkward sentences. Because GVLMs rely heavily on their internal LLM decoders, they naturally assign higher probabilities to the grammatically correct positive captions purely because they sound better, completely ignoring the visual content of the image.
Methods
The authors propose a two-pronged approach to expose and mitigate this syntactical shortcut:
- SyntaxBias Score: To quantify how much a benchmark relies on grammatical tells, the authors use a strong standalone LLM (Vicuna-13B) to evaluate the text pairs without providing the image. If the LLM can easily guess the correct caption based solely on text fluency, the dataset has a high syntactical bias.
- The SADE Benchmark (SyntActically DE-biased): The authors take existing benchmarks and filter out examples with high SyntaxBias Scores.
- Content-Only Challenge: To truly test visual grounding, they introduce a new task. They strip the positive captions of all grammar, leaving only a raw list of nouns and adjectives (e.g., "woman, peach tank top, mountain bike"). They pair this with negative captions that are perfectly fluent, grammatically flawless sentences, but completely unrelated to the image. To succeed, the model must overcome its natural bias toward syntactic fluency and pick the ungrammatical, visually accurate option.
Datasets
The authors evaluated and filtered four major existing compositionality benchmarks:
- Winoground
- VL-Checklist
- ARO
- CREPE
By filtering these and adding the new Content-Only Challenge derived from COCO and Flickr30K, they compiled the SADE benchmark, totaling over 52,000 images and nearly 130,000 reference sentences.
Key Insights and Ideas
- The Blind LLM Shortcut: The most striking finding comes from an ablation study where the authors replaced the evaluation images with pure random noise. While encoder-based models like CLIP plummeted to random-guess accuracy, GVLMs barely saw a performance drop on existing benchmarks. They were solving the visual puzzles completely blind!
- Fluency Traps: GVLMs prioritize syntactic correctness over visual truth. As shown in their Flickr30K experiments, if a positive caption is ungrammatical, the model's accuracy drops significantly because it prefers a fluent, completely fabricated negative caption.
Key Results
The authors benchmarked several state-of-the-art GVLMs on SADE:
- InstructBLIP and LLaVA-13B emerged as the most robust models, achieving the highest scores on the difficult Content-Only Challenge (57.83% and 49.80% respectively).
- Emu demonstrated a massive vulnerability. While it scored an impressive 85.84% on standard attribute testing, its performance collapsed to a abysmal 2.79% on the Content-Only Challenge. This indicates that Emu is highly sensitive to grammatical fluency and easily tricked by the "fluency trap."
Limitations and Future Work
The authors note two primary limitations:
- Dataset Scale: Because SADE relies on filtering out biased samples from existing datasets, the final benchmark scale is relatively small, which may limit its generalization.
- Scope of Evaluation: SADE is strictly focused on visio-linguistic compositionality (objects, attributes, relations). It does not evaluate other crucial GVLM dimensions, such as multi-turn dialogue capability or complex multi-step reasoning.
Future work will likely focus on generating larger, natively de-biased datasets rather than relying on filtering existing ones.
Summary updated