Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

Eun Woo Im, Dhruv Madhwal, Vivek GuptaarXiv 2026 · arXiv

TL;DR

Vision-Language Models (VLMs) like CLIP frequently exhibit "bag-of-words" behavior, failing to grasp word order and attribute binding due to a lack of high-quality negative examples during pretraining. This paper introduces Slipform, a framework that leverages psycholinguistic lexical concreteness to systematically generate visually distinct hard negatives and employs a novel Cement loss with an adaptive margin to fix severe gradient imbalances during contrastive training. The result is a state-of-the-art boost in multimodal compositional understanding without sacrificing general visual representation capabilities.

What Problem Does This Paper Address?

While contrastive VLMs excel at matching global summaries of scenes, they stumble on fine-grained compositional reasoning—such as distinguishing "a dog chasing a cat" from "a cat chasing a dog," or correctly binding attributes to objects.

Standard contrastive learning relies on random batch sampling, which yields "easy negatives" that are too simple to force the model to learn structural nuances. While hard negative mining (generating slightly altered captions and images) helps, existing methods lack a principled way to choose which words to alter. For instance, changing an abstract modifier like "contemporary bedroom" to "old-style bedroom" results in negligible visual changes. This provides a weak learning signal for the VLM.

Furthermore, the authors reveal a hidden flaw in the standard InfoNCE loss: as batch sizes scale up, the sheer volume of easy negatives creates a massive gradient imbalance, drowning out the critical learning signals from hard negatives.

Examples of abstract vs concrete word perturbations.

Methods

The authors introduce Slipform, an integrated framework consisting of two core components:

  1. ConcretePlant (Automated Hard Negative Generation): Instead of randomly altering words, this pipeline targets words with high lexical concreteness (how strongly a concept is tied to perceptible physical entities). It parses a caption using SpaCy, looks up psycholinguistic concreteness scores from a standardized database, and selects highly concrete target words via Top-K sampling. An LLM (Qwen3-32B) perturbs the text, and a generative model (SDXL-Turbo) edits the original image to reflect this specific structural change.
Overview of the ConcretePlant pipeline.
  1. Cement Loss (Concreteness-Enhanced Margin Estimation): To prevent easy negatives from overwhelming the optimization process, the authors introduce an adaptive margin ($\hat{m}_i$) injected directly into the hard negative similarity term of the contrastive loss function. Because low-concreteness words produce noisier, less visually distinct modifications, the margin is dynamically scaled using a Fermi-Dirac distribution. Highly concrete, reliable training pairs receive a larger margin to amplify their learning signal, while ambiguous, low-concreteness pairs are suppressed.

Datasets

  • Training: The framework was trained on the MS-COCO Karapathy train split to generate a custom dataset called ConcreteBatch. To validate their hypothesis, they curated three controlled subsets: $D_{hc}$ (high concreteness), $D_{lc}$ (low concreteness), and $D_{wo}$ (randomly sampled/without concreteness control).
  • Evaluation:
  • Compositional Understanding: SugarCrepe, SugarCrepe++, and Winoground.
  • General Visual Representation: ImageNet-1k (single-label linear probing), MS-COCO (multi-label linear probing), Flickr30k (zero-shot cross-modal retrieval), and the VTAB benchmark (19 subcategories).

Key Insights and Ideas

  • Concreteness Dictates Efficacy: Modifying highly concrete terms (e.g., swapping "chicken" for "beef") forces generative models to make localized, distinct physical changes rather than ambiguous global style transformations.
  • The InfoNCE Gradient Explosion: The authors empirically show that easy negatives consume at least 72% of the total gradient signal when the batch size reaches $N=1024$. Shrinking the batch size to fix this degrades overall performance, proving that an architectural or loss-based fix (like Cement loss) is required.
Gradient magnitude controlled by margin across training steps.
  • Logit Gaps Align with Psychology: Empirical analysis showed that the model's logit gaps correlate linearly with human-derived psycholinguistic concreteness scores, providing a solid theoretical foundation for using the Fermi-Dirac distribution to model the adaptive margin.

Key Results

  • State-of-the-Art Compositionality: Slipform achieved a 13.13% relative gain in macro-accuracy over the standard CLIP baseline across compositional benchmarks, outperforming competitive hard-negative frameworks like NegCLIP and DeGLA with a macro average score of 54.18%.
  • Mitigating the Trade-Off: Hard negative training typically triggers a trade-off where fine-grained compositional gains come at the cost of general downstream classification performance. Thanks to the adaptive margin in Cement loss, Slipform successfully mitigates this, maintaining highly competitive performance on ImageNet-1k (43.11% top-1 linear probing) and Flickr30k retrieval.

Limitations and Future Work

The authors note a few areas for improvement:

  • Multimodal Concreteness Discrepancies: Standard psycholinguistic concreteness scores account for all human senses (tactile, auditory, olfactory), not just vision. Incorporating purely visual concreteness metrics could refine the pipeline.
  • Generalization vs. Compositionality: While Cement loss successfully cushions the blow, a natural tension still exists between optimizing for fine-grained discrimination and broader visual representation objectives. Future work will explore training factors to completely bridge this gap.

Summary updated

Abstract

Vision-Language Models demonstrate remarkable capabilities but often struggle with compositional reasoning, exhibiting vulnerabilities regarding word order and attribute binding. This limitation arises from a scarcity of informative samples needed to differentiate subtle semantic variations during contrastive pretraining. Although hard negative mining offers a promising remedy, existing methods lack explicit mechanisms to dictate which linguistic elements undergo modification. Instead of engineering generative architectures, this study establishes lexical concreteness as a fundamental determinant of negative sample efficacy. Modifying highly concrete terms generates more pronounced structural and visual discrepancies, providing a substantially stronger learning signal. Leveraging this principle, ConcretePlant is proposed to systematically isolate and manipulate perceptually grounded concepts. Analyses of the InfoNCE further reveals a severe gradient imbalance, where easily distinguishable pairs disproportionately overwhelm the optimization process and restrict the bandwidth available for nuanced learning. To resolve this degradation, the Cement loss is formulated utilizing a margin-based approach. By correlating psycholinguistic scores with sample difficulty, this objective dynamically calibrates the penalization applied to individual training pairs. Comprehensive evaluations substantiate these theoretical claims. The integrated framework, designated as Slipform, achieves state-of-the-art accuracy across diverse compositional evaluation benchmarks, general cross-modal retrieval, single and multi label linear probing.
BibTeX
@misc{im2026concrete,
  author = {Eun Woo Im and Dhruv Madhwal and Vivek Gupta},
  title = {{Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding}},
  year = {2026},
  eprint = {2604.13313},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.13313},
}
Figure