TL;DR
Vision-Language Models (VLMs) like CLIP frequently exhibit "bag-of-words" behavior, failing to grasp word order and attribute binding due to a lack of high-quality negative examples during pretraining. This paper introduces Slipform, a framework that leverages psycholinguistic lexical concreteness to systematically generate visually distinct hard negatives and employs a novel Cement loss with an adaptive margin to fix severe gradient imbalances during contrastive training. The result is a state-of-the-art boost in multimodal compositional understanding without sacrificing general visual representation capabilities.
What Problem Does This Paper Address?
While contrastive VLMs excel at matching global summaries of scenes, they stumble on fine-grained compositional reasoning—such as distinguishing "a dog chasing a cat" from "a cat chasing a dog," or correctly binding attributes to objects.
Standard contrastive learning relies on random batch sampling, which yields "easy negatives" that are too simple to force the model to learn structural nuances. While hard negative mining (generating slightly altered captions and images) helps, existing methods lack a principled way to choose which words to alter. For instance, changing an abstract modifier like "contemporary bedroom" to "old-style bedroom" results in negligible visual changes. This provides a weak learning signal for the VLM.
Furthermore, the authors reveal a hidden flaw in the standard InfoNCE loss: as batch sizes scale up, the sheer volume of easy negatives creates a massive gradient imbalance, drowning out the critical learning signals from hard negatives.
Methods
The authors introduce Slipform, an integrated framework consisting of two core components:
- ConcretePlant (Automated Hard Negative Generation): Instead of randomly altering words, this pipeline targets words with high lexical concreteness (how strongly a concept is tied to perceptible physical entities). It parses a caption using SpaCy, looks up psycholinguistic concreteness scores from a standardized database, and selects highly concrete target words via Top-K sampling. An LLM (Qwen3-32B) perturbs the text, and a generative model (SDXL-Turbo) edits the original image to reflect this specific structural change.
- Cement Loss (Concreteness-Enhanced Margin Estimation): To prevent easy negatives from overwhelming the optimization process, the authors introduce an adaptive margin ($\hat{m}_i$) injected directly into the hard negative similarity term of the contrastive loss function. Because low-concreteness words produce noisier, less visually distinct modifications, the margin is dynamically scaled using a Fermi-Dirac distribution. Highly concrete, reliable training pairs receive a larger margin to amplify their learning signal, while ambiguous, low-concreteness pairs are suppressed.
Datasets
- Training: The framework was trained on the MS-COCO Karapathy train split to generate a custom dataset called ConcreteBatch. To validate their hypothesis, they curated three controlled subsets: $D_{hc}$ (high concreteness), $D_{lc}$ (low concreteness), and $D_{wo}$ (randomly sampled/without concreteness control).
- Evaluation:
- Compositional Understanding: SugarCrepe, SugarCrepe++, and Winoground.
- General Visual Representation: ImageNet-1k (single-label linear probing), MS-COCO (multi-label linear probing), Flickr30k (zero-shot cross-modal retrieval), and the VTAB benchmark (19 subcategories).
Key Insights and Ideas
- Concreteness Dictates Efficacy: Modifying highly concrete terms (e.g., swapping "chicken" for "beef") forces generative models to make localized, distinct physical changes rather than ambiguous global style transformations.
- The InfoNCE Gradient Explosion: The authors empirically show that easy negatives consume at least 72% of the total gradient signal when the batch size reaches $N=1024$. Shrinking the batch size to fix this degrades overall performance, proving that an architectural or loss-based fix (like Cement loss) is required.
- Logit Gaps Align with Psychology: Empirical analysis showed that the model's logit gaps correlate linearly with human-derived psycholinguistic concreteness scores, providing a solid theoretical foundation for using the Fermi-Dirac distribution to model the adaptive margin.
Key Results
- State-of-the-Art Compositionality: Slipform achieved a 13.13% relative gain in macro-accuracy over the standard CLIP baseline across compositional benchmarks, outperforming competitive hard-negative frameworks like NegCLIP and DeGLA with a macro average score of 54.18%.
- Mitigating the Trade-Off: Hard negative training typically triggers a trade-off where fine-grained compositional gains come at the cost of general downstream classification performance. Thanks to the adaptive margin in Cement loss, Slipform successfully mitigates this, maintaining highly competitive performance on ImageNet-1k (43.11% top-1 linear probing) and Flickr30k retrieval.
Limitations and Future Work
The authors note a few areas for improvement:
- Multimodal Concreteness Discrepancies: Standard psycholinguistic concreteness scores account for all human senses (tactile, auditory, olfactory), not just vision. Incorporating purely visual concreteness metrics could refine the pipeline.
- Generalization vs. Compositionality: While Cement loss successfully cushions the blow, a natural tension still exists between optimizing for fine-grained discrimination and broader visual representation objectives. Future work will explore training factors to completely bridge this gap.
Summary updated