Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples

Yeyuan Wang, Dehong Gao, Lei Yi, Linbo Jin, Jinxia Zhang, Libin Yang, Xiaoyan CaiAAAI 2025 · DOI

Introduction

Vision-Language Pretraining (VLP) has revolutionized how machines understand the relationship between images and text. Models like CLIP and ALBEF have demonstrated remarkable capabilities in matching general captions to visual scenes. However, a significant hurdle remains: fine-grained understanding. While these models can easily identify "a dog in a park," they frequently stumble when asked to distinguish subtle spatial relations (e.g., "the cat is on the mat" vs. "the cat is under the mat"), specific attributes, or complex compositional logic.

This paper introduces Negative Augmented Samples (NAS), a novel pretraining strategy designed to bridge the "semantic gap" between continuous pixel data and discrete linguistic tokens. By treating images more like language through a visual dictionary, NAS forces models to move beyond a "bag of pixels" approach and develop a nuanced appreciation for detail.

The Semantic Gap and Fine-Grained Challenges

The core challenge in VLP is that language is composed of discrete, symbolic tokens, whereas images are continuous streams of pixels. Most existing models align these modalities by extracting holistic, high-level features. While effective for broad classification, this approach ignores local, nuanced interactions.

Furthermore, while "hard negative" techniques—training a model to distinguish a correct caption from a slightly incorrect one—are common in the text domain (Negative Textual Augmentation), they are difficult to implement for images. Editing an image semantically without breaking its visual consistency or creating obvious artifacts is a complex task. NAS addresses this by discretizing the visual space.

Methodology: The NAS Framework

The NAS framework introduces two primary innovations to facilitate fine-grained alignment: the Visual Dictionary and Negative Visual Augmentation.

1. The Visual Dictionary (VD)

To bridge the gap between pixels and words, the model maintains a dictionary of visual feature vectors. During training, image patches are "quantized"—mapped to the nearest entry in this dictionary. This process transforms a continuous image into a sequence of discrete visual tokens, effectively creating a "visual language."

Vector Quantization process converting an image into discrete visual tokens.

2. Negative Visual Augmentation (NVA)

Once the image is tokenized, the model can perform precise semantic edits. The NVA process follows a specific pipeline:

  • Primary Object Identification: The model identifies which visual tokens are most relevant to the global text and image features (the "primary object").
  • Token Replacement: It replaces approximately 30% of these object-related tokens with the "top-k" most similar, yet distinct, tokens from the Visual Dictionary.
  • Hard Negative Creation: This results in a "hard negative" image that is visually similar to the original but contains subtle semantic errors (e.g., swapping a butterfly for a bee in a specific patch).
The full Negative Visual Augmentation (NVA) pipeline, showing quantization followed by semantic token replacement to create a hard negative.

The model is then trained using Fine-Grained Image-Text Matching (FGITM), where it must distinguish the original pair from these subtly corrupted versions. This forces the encoder to pay much closer attention to specific visual details and their alignment with text.

Experimental Evaluation

The researchers pretrained the model on a standard corpus of 4 million image-text pairs (including MSCOCO, Visual Genome, SBU Captions, and CC-3M). To test fine-grained understanding, they utilized several rigorous benchmarks:

  • ARO (Attributes, Relations, and Order): Evaluates the model's grasp of how objects relate to one another.
  • Winoground: A challenging compositional benchmark where two images and two captions share identical words but differ in meaning due to word order.
  • VALSE: Focuses on specific linguistic phenomena such as existence, counting, and coreference.

Key Results and Insights

NAS (specifically the version combining both visual and textual negative augmentation, NTVA) achieved state-of-the-art (SOTA) results across all major fine-grained benchmarks:

  • Massive Gains on ARO: The model improved over the ALBEF baseline by 18.8%, a significant leap in relational understanding.
  • Superior Compositionality: On Winoground, NAS outperformed ALBEF by 7.2% and surpassed much larger models like LLaVA-7B.
  • Linguistic Precision: It achieved a new SOTA average score of 74.4 on VALSE, showing particular strength in identifying the existence of objects and performing accurate counts.

Qualitative analysis showed that while baseline models often failed to distinguish between "sheep standing" and "no sheep," or struggled to count "5 birds," the NAS-trained model correctly identified these nuances.

Conclusion and Future Work

The NAS framework demonstrates that discretizing the visual space through a Visual Dictionary makes image manipulation as "programmable" as text. This symmetry in augmentation—treating both images and text as sequences of discrete tokens—is key to achieving fine-grained vision-language alignment.

While currently implemented on the ALBEF framework, future research will explore scaling this approach to decoder-only Large Multi-modal Models (LMMs) and optimizing the granularity of the Visual Dictionary to ensure it captures the "Goldilocks" zone of detail—neither too coarse to be vague nor too fine to be noisy.

Summary updated

BibTeX
@inproceedings{wang2025enhancing,
  author = {Yeyuan Wang and Dehong Gao and Lei Yi and Linbo Jin and Jinxia Zhang and Libin Yang and Xiaoyan Cai},
  title = {{Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32869},
}
Figure