Vision-language models such as CLIP score an image against a caption by comparing two summary vectors. They often behave like a "bag of words": a caption scores well if its words appear somewhere in the image, whether or not they are bound together correctly. The most effective recent fixes break the caption into parts. Each entity, attribute, and relation is checked against a region of the image, and the local scores are combined. This note argues that these pipelines carry a hidden flaw that logic programming identified decades ago: circular justification. Circular justification happens when two claims are accepted only because each supports the other. Suppose a detector finds "the rider" partly because it was also asked about "the horse", and the horse region is confirmed partly because a rider sits on it. Then both claims can pass verification even though neither has independent visual evidence. Answer Set Programming, a logic-programming formalism whose semantics rejects exactly such self-supporting sets (called unfounded sets), has precise and cheap tools for forbidding this pattern. These tools are loop formulas, which require every cycle of mutually supporting claims to have at least one member grounded from outside the cycle.
The proposal is to build an image-caption matcher that computes the best-scoring set of local grounding claims that is founded. In a founded set, context may help localize an object, but every chain of contextual support must bottom out in at least one piece of context-free visual evidence. This sits between two designs that exist today. One is strict fixed-order pipelines, which forbid contextual help and so lose small or occluded objects. The other is joint scoring, which allows mutual support and so accepts captions whose parts only confirm each other. The claim is testable. Our matcher should keep the recall benefits of contextual grounding while refusing captions about objects that are not there. The decisive test is a new set of controlled "partner-removed" images.
Core Claim
Circularity in decomposed matching has a concrete source. A phrase-conditioned detector is given the whole caption, or a phrase containing a relation, and chooses a region. The same or a correlated model then scores that region against the phrase. Selection and verification share evidence, so the check confirms itself. Max-over-crops aggregation, as in the inference-time approach Adding simple structure at inference improves Vision-Language Compositionality, has the same shape: each text segment picks its own best crop independently.
We claim that making each image-caption pair into a tiny logic program removes a specific, measurable class of false acceptances. In that program, atoms are grounding claims and rules record which evidence each claim used. Requiring the accepted claims to form a stable (founded) model removes these false acceptances, and it does so without the recall loss of forbidding context altogether.
The logic machinery is reused, not invented. ASSAT: Computing Answer Sets of a Logic Program by SAT Solvers supplies the loop formula and its lazy, add-when-violated use. A Model-Theoretic Counterpart of Loop Formulas supplies the characterization that a model is stable exactly when every loop in it has external support. Default Logic, Propositional Logic, and Constraints supplies an equivalent encoding with integer levels, which we use for the weighted, uncertain version. None of these works handles continuous perceptual scores. Adapting them to scores is the technical step.
Research Program
For a caption, a parser extracts entities, attributes, and subject-relation-object triples, in the style of Structure-CLIP's scene-graph parsing and GroundingCLIP's LLM decomposition. For each entity $e$ we generate two kinds of candidate regions.
The first kind is context-free proposals. These come from a detector prompted with the bare noun only ("horse"). Each has a score from the vision-language model on the tight crop.
The second kind is contextual proposals. These come from a detector prompted with a phrase mentioning another entity ("horse under a rider"). Each such proposal is recorded as depending on the grounding of that other entity.
Attribute claims are verified on the chosen region. Relation claims are verified on the union region of the two entities, with both boxes marked. This keeps the spatial reference frame that ITA loses when it crops objects in isolation.
The result is a small weighted program. Each grounding atom $g$ has a score $s(g)$, and rules state which atoms each contextual atom depends on. We define the match score as
$$S(I, C) = \max_{M \in \mathcal{F}(I,C)} \sum_{g \in M} s(g) \; - \; \lambda \cdot \#\text{unmatched}(C, M),$$
where $I$ is the image, $C$ the caption, $\mathcal{F}(I,C)$ the set of founded assignments that give each caption element at most one grounding, $\#\text{unmatched}$ counts caption elements left ungrounded, and $\lambda$ is a single penalty tuned on a held-out split.
Foundedness is enforced by the loop formula. For any cycle $L$ of contextual atoms, the formula states
$$\bigvee_{g \in L} g \;\Rightarrow\; \bigvee \mathrm{ExtSupport}(L),$$
where $\mathrm{ExtSupport}(L)$ are the rules that derive a member of $L$ from atoms outside $L$, such as a context-free proposal. Without the loop formula, the maximization is ordinary joint scoring. With it, a pair of claims that only vouch for each other is dropped. Following ASSAT, we solve without loop formulas first, check the returned assignment for unfounded cycles, add only the violated formulas, and re-solve.
Captions rarely have more than five or six entities, so these programs are tiny and solve in milliseconds. Known worst-case explosions in the number of loops do not bite here. As a cross-check, we also implement the Ben-Eliyahu–Dechter level encoding as a small integer program, which requires $\text{level}(g) > \text{level}(h)$ whenever $g$ is derived from $h$. The two encodings should return identical scores. Where they do not, the difference exposes bugs in the dependency bookkeeping.
The program has an analysis half as well. We instrument reimplementations of GroundingCLIP Compositional Image-Text Matching and Retrieval by Grounding Entities and ITA to log which evidence produced each accepted local match. From these logs we compute a circular acceptance rate: the fraction of accepted captions whose supporting claims contain an unfounded cycle. This analysis matters in its own right. The research file raises circular grounding only as a hypothesis, and no paper in it measures how often it happens. If the rate is near zero on real benchmarks, that is a publishable negative finding, and it would narrow the method's claim to controlled settings.
Baselines
The naive reference point is the backbone alone (CLIP ViT-B/32 and SigLIP 2), scoring the global image embedding against the global caption embedding. On Winoground the relevant floor is the 16.67% random Group Score, which models like VinVL (14.50%) and FLAVA (14.25%) fail to beat, as reported in Winoground.
The strongest training-free competitor is ITA Adding simple structure at inference improves Vision-Language Compositionality. It matches hierarchical text segments against 86 to 270 multi-scale crops and averages the top similarities. It is the strongest because it has the largest reported gains on the swap benchmarks we target. It raises CLIP's BiSCoR-CTRL Group Score from 1.4 to 24.8 and SigLIP 2-Giant's BiVLC Group Score from 7.8 to 29.8. We run it on the same backbones. We also cap our method's total vision-model forward passes at ITA's crop budget, so any win cannot come from extra compute. Beating it means a higher Group Score on BiSCoR-CTRL and BiVLC at equal passes, with the largest margin on the Size and spatial subsets that ITA reports as weak.
The strongest grounding-based competitor is GroundingCLIP, which grounds LLM-extracted entities with a detector and fuses local and global scores. It reports a 12% Recall@1 improvement over CLIP on Flickr30K. We reimplement it with the identical parser, detector, and backbone we use, so the only difference is how local claims are combined.
The strongest LLM-based competitor is CECE Natural Language Inference Improves Compositionality in Vision-Language Models. It expands captions into entailments and contradictions and scores each with the vision-language model. It reports +19.2% Winoground Group Score over prior best and +12.9% on EqBen. The file gives no absolute value, so we will rerun it with the same LLM we use for parsing.
As trained references, we report LABCLIP CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally, which reaches 95% on CLEVR, and CLIC Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning, which reports +9% on SugarCrepe++ image-to-text. Both use their released checkpoints. Our method can also run on top of either one.
The two ablations that isolate the key ingredient share every component with our method and change only the support rule. The unconstrained joint variant maximizes the same score with the loop formulas removed, so mutual support is allowed. The fixed-order variant allows only context-free proposals, so contextual help is forbidden. The founded method must beat unconstrained joint on false acceptances and beat fixed-order on recall. Otherwise the loop-formula idea adds nothing.
Why not just…?
Why not just use a fixed grounding order (nouns first, then attributes, then relations)? This is the most serious threat. A fixed order is trivially acyclic and needs no solver. On clean benchmarks where every entity is easy to detect alone, we expect it to tie with our method. The founded matcher differs only when context-free detection fails and contextual detection succeeds: small, occluded, or ambiguous entities. We will therefore report results stratified by context-free detection confidence. If the fixed-order variant matches ours across all strata, we will concede that loop formulas are unnecessary and report the circular-acceptance analysis as the main contribution.
Why not just add more crops, as ITA does? ITA's own results show the limit. Isolated crops strip the reference frame needed for Size and spatial predicates, and independent max-over-crops lets each phrase pick its own region, which is itself a self-selection effect. Our union-region relation check keeps the reference frame. The partner-removed test below directly measures whether independent crop selection accepts captions about missing objects.
Why not just let an LLM generate entailments, as CECE does? CECE improves Winoground substantially. But its checks are scored by the same global model and chosen by unconstrained prompting, so nothing prevents two generated entailments from being "confirmed" by the same misleading context. CECE is complementary to our method: its entailments can be added as extra claims inside the founded program. We will test CECE alone, ours alone, and the combination.
Why not just fine-tune on hard negatives, as CLIC or LABCLIP do? Hard-negative training improves binding. But SugarCrepe showed that NegCLIP's gains partly came from text artifacts. LABCLIP reportedly stays weak on spatial relations and pays a small ImageNet cost. Training also does not address whether a match is supported by present evidence. We run our method on top of the CLIC and LABCLIP checkpoints to see whether the gains stack or overlap.
Proposed Experiments
The main evaluation covers Winoground (Text, Image, and Group Scores), SugarCrepe with emphasis on the SWAP subsets, where GPT-4V reaches only 83.13% against roughly 99% for humans, BiSCoR-CTRL and BiSCoR-NAT, BiVLC, and ARO VG-Relation. Flickr30K and MS-COCO Recall@1 check that retrieval is not taxed.
The decisive check is a new partner-removed test set. We start from CLEVR/PUG-style synthetic scenes and COCO images with relational captions. In each image we remove one entity of a relation, by re-rendering synthetic scenes or inpainting COCO images, while keeping the remaining entity's pose or placement, for example a riding posture with the horse erased. The original caption is then false.
A method vulnerable to circular support should keep accepting these captions at a rate well above its rate on unrelated negatives. The founded method should reject them at close to the fixed-order rate. It should also retain the unconstrained joint variant's accuracy on the unmodified, occluded-entity images.
We report three quantities for every method on the same images: the circular acceptance rate, false acceptance on partner-removed pairs, and recall on hard-to-detect entities. The hypothesis fails if the unconstrained joint variant shows no excess false acceptance on partner-removed images. In that case circular support is not a real failure mode of current pipelines, and the loop-formula constraint is unnecessary.
References
- Adding simple structure at inference improves Vision-Language Compositionality
- ASSAT: Computing Answer Sets of a Logic Program by SAT Solvers
- A Model-Theoretic Counterpart of Loop Formulas
- Default Logic, Propositional Logic, and Constraints
- Compositional Image-Text Matching and Retrieval by Grounding Entities
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
- Natural Language Inference Improves Compositionality in Vision-Language Models
- CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
- Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning
- SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality