Introduction
Compositionality—the capacity to understand how discrete objects, attributes, and relations combine to form complex semantic structures—is a fundamental pillar of human intelligence. While early encoder-based vision-language models like CLIP were often criticized for behaving as "bags-of-words" that ignored spatial and relational syntax, the advent of Large Generative Vision-Language Models (GVLMs) such as LLaVA promised a new era of reasoning. These models leverage powerful Large Language Model (LLM) backbones to bridge visual and textual understanding. However, recent investigations suggest that their apparent reasoning capabilities may be partially illusory, driven by linguistic priors rather than genuine visual grounding.
The Syntactical Bias Problem
A critical flaw identified in current GVLM evaluation is Syntactical Bias. Most existing benchmarks utilize "hard negatives" created by shuffling words into ungrammatical sequences. Because GVLMs are built upon LLMs trained to maximize text fluency, they naturally assign higher probabilities to well-formed sentences regardless of their alignment with the visual input. Consequently, a model might appear to "solve" a compositionality task simply because the correct answer is a fluent sentence while the distractor is gibberish. This paper quantifies this over-reliance and proposes a more rigorous evaluation framework.
Methodology
VisualGPTScore and Metric Sensitivity
The researchers first analyzed the standard evaluation metric for GVLMs: the VisualGPTScore (the log-likelihood of a caption given an image). They demonstrated that this metric is disproportionately sensitive to text fluency. As shown in Figure 3, when comparing various metrics across different challenge levels, generative scores exhibit distinct distribution patterns that reflect their sensitivity to linguistic structure over visual content.
SyntaxBias Score
To quantify this phenomenon, the authors introduced the SyntaxBias Score. This metric utilizes a standalone LLM (e.g., Vicuna-13B) to calculate the probability of a sentence without an accompanying image. If a "correct" sentence is significantly more probable than its "negative" counterpart based on text alone, the sample is flagged as biased, as the model can pass the test without ever processing the visual data.
The SADE Benchmark
To address these vulnerabilities, the researchers developed the SADE (SyntActically DE-biased) benchmark. SADE improves upon existing suites like ARO and VL-CheckList through two primary mechanisms:
- Filtering: Removing samples with high SyntaxBias Scores to ensure models cannot rely solely on linguistic fluency.
- Content Challenge: Introducing a task where the positive reference is a syntax-free list of keywords, while the negatives are fluent, grammatically perfect sentences that are factually unrelated to the image. This forces the model to prioritize visual alignment over grammatical preference.
Experimental Analysis
The "Noisy Image" Experiment
In a definitive test of visual grounding, the researchers replaced benchmark images with random noise. While CLIP’s performance dropped significantly, LLaVA’s performance on standard benchmarks remained remarkably stable. This "smoking gun" confirms that GVLMs frequently solve these benchmarks using internal linguistic patterns rather than visual-textual grounding.
Benchmark Vulnerability
Empirical results highlight the fragility of current models. On standard benchmarks with scrambled negatives, LLaVA-7B achieved an impressive 98.62% accuracy. However, when faced with fluent but unrelated negatives, accuracy plummeted to 67.06%. In the most difficult scenario—the Content Challenge—accuracy fell to a mere 27.02%, indicating that the models would rather select a fluent, incorrect sentence than a correct, non-syntactic list of keywords.
Results and Discussion
Evaluation on the SADE benchmark revealed a significant performance gap across all tested GVLMs. While models like InstructBLIP and Emu showed greater robustness in dimensions such as Relation and Attribute understanding, even the top performers struggled with the Content Challenge. This suggests that the LLM backbone acts as a double-edged sword: providing "common sense" that often masks a failure to accurately perceive the visual input.
Conclusion
This examination reveals that GVLMs are currently "syntax-worshippers," prioritizing linguistic fluency over visual truth. The SADE benchmark provides a more honest yardstick for measuring true compositionality. Future work must focus on developing architectures that more tightly couple visual features with the LLM's latent space, ensuring that the model's linguistic "comfort zone" does not override its visual perception.
Summary updated