The Compositional Grounding Gap: Why Vision-Language Models Fail at Relational Reasoning and How to Fix It

Kaustubh S. BukkapatnamALVR @ ACL 2026 · OpenReview

Abstract

Large vision-language models (LVLMs) achieve strong performance on many multimodal tasks, yet consistently fail at compositional relational reasoning—distinguishing "the cat on the mat" from "the mat on the cat." We provide a formal explanation for this failure. We prove that any vision-language alignment operating on pooled (order-invariant) visual features contains compositional blind spots: semantically distinct scenes that map to identical representations. We show that the number of blind spots grows factorially with scene complexity, establishing a fundamental limit on pooled-feature architectures. Motivated by this analysis, we propose REGROUND, a training-free, test-time method that re-introduces spatial structure into alignment by performing relation-guided cross-attention over spatial visual tokens, directed by a lightweight parse of the text query. Without any fine-tuning, REGROUND improves compositional accuracy by +8.6 points on Winoground, +8.4 on ARO-Relation, +6.4 on SugarCrepe, and +8.4 on VSR when applied to LLaVA-1.5, and provides consistent gains across other LVLMs. Ablation studies confirm that each component—parse guidance, token-level attention, and relation masking—contributes significantly.
BibTeX
@inproceedings{bukkapatnam2026compositional,
  author = {Kaustubh S. Bukkapatnam},
  title = {{The Compositional Grounding Gap: Why Vision-Language Models Fail at Relational Reasoning and How to Fix It}},
  booktitle = {ALVR @ ACL},
  year = {2026},
  url = {https://openreview.net/forum?id=IMzDg3kcYu},
}

Loading related content...

Figure