TL;DR
Vision-language datasets suffer from reporting bias: people rarely describe obvious commonsense details (like grass being green or a dog being brown), causing models to latch onto spurious object–attribute correlations. The authors introduce BiAug, a pipeline that uses large language models (LLMs) and diffusion inpainting to decouple objects from their default attributes, synthesizing aligned image–text pairs and cross-modal hard negatives. Fine-tuning CLIP with BiAug boosts compositionality and zero-shot image-text retrieval without requiring new human annotations.
What Problem Does This Paper Address?
When humans write captions, they tend to omit details that feel like common sense. A caption might say "a dog runs with a tennis ball," omitting whether the dog is brown or black, or whether it runs on snow or grass. This tendency is known as reporting bias.
For multimodal foundation models like CLIP, reporting bias is toxic:
- Ambiguous pairings: The exact same caption can match visually disparate images.
- Spurious feature binding: Models overfit to dominant co-occurrences (e.g., assuming a "lawn" is always green, or associating "flag" almost exclusively with the American flag), failing to decouple objects from their attributes.
- Bag-of-words behavior: Vision-language models often struggle to tell the difference between "a white road and a paved house" vs. "a paved road and a white house."
Existing dataset-cleaning works focus either on caption bootstrapping or image-only generation, but neither directly fixes the joint image–text reporting bias.
Methods
To break the tight coupling between objects and their frequent attributes, BiAug (Bimodal Augmentation) applies a three-phase pipeline:
Source Pair ──► [Phase I: LLM + GroundingDINO] ──► Object Detection & Visual Filtering
│
▼
[Phase II: LLM Decoupling] ──► Generate Diverse & Hard Negative Attributes
│
▼
[Phase III: Stable Diffusion] ──► Mask & Inpaint Contrasting Attributes
│
▼
Augmented Bimodal Hard-Negative Pairs
- LLM-Aided Object Detection: Given an initial caption, an LLM (
gpt-3.5-turbo) infers up to five objects likely present in the scene—especially unmentioned background objects. GroundingDINO then grounds and verifies these candidates on the image, filtering out false positives and low-confidence detections. - Object–Attribute Decoupling via LLM: For each validated object, the LLM generates descriptive attributes spanning four categories (color, shape, material, and other). Crucially, it also generates an opposing "hard negative" attribute (e.g., “lush green grass” vs. “dry grass”), inserting both into the original caption to produce paired contrasting captions.
- Image Inpainting & Negative Construction: Using bounding boxes from Phase 1, the object area is masked. A pre-trained diffusion inpainter (Stable Diffusion Inpainting) modifies the region according to the contrasting descriptions.
This creates balanced, bimodal hard-negative triplets where the text and the visual appearance vary strictly across specific attribute dimensions. The model (a standard CLIP ViT-B/16) is fine-tuned using contrastive ranking loss with these samples treated as in-batch negatives.
Datasets
- Training/Augmentation: Subsets of Conceptual Captions 3M (CC3M) scaled at 40K, 100K, 200K, and 300K samples. CC3M was selected because its captions naturally use hypernyms with well-defined attribute spaces.
- Compositionality Evaluation: The ARO (Attribution, Relation, and Order) benchmark (~50k test cases), focusing on:
- VG-Attribute and VG-Relation (swapped objects and attributes).
- Flickr30K-Order and COCO-Order (word/phrase order scrambling).
- Standard Retrieval Evaluation: Karpathy test splits of MSCOCO and Flickr30K.
Key Insights and Ideas
- Omission is widespread: In human evaluation, ~60% of the objects proposed by the LLM and grounded by the detector were completely absent from the original caption, confirming that text-image pairs systematically omit visual context.
- BiAug stabilizes scaling: When fine-tuning vanilla CLIP (
CLIP-ft) on larger amounts of raw CC3M data, performance on compositional benchmarks (ARO) actually degraded due to reinforcement of biased shortcuts. In contrast, BiAug continuously improved as more augmented data was added. - Filtering is non-negotiable: Inpainting tools are imperfect (human evaluators judged inpainting attribute correctness at ~63%). Bounding-box thresholding (rejecting boxes under 5% or over 80% of image size, or with >70% overlap) and confidence filtering are essential; removing filtering caused severe performance drops.
Key Results
- Compositionality (ARO Benchmark, 300K subset):
- VG-Relation Macro Accuracy: BiAug reached ~69.3%, outperforming standard fine-tuning (~67.0%) and zero-shot CLIP (~63.7%).
- VG-Attribute Macro Accuracy: BiAug achieved ~64.0%, whereas standard fine-tuning plateaued and dropped to ~62.4%.
- Cross-Modal Retrieval (Karpathy Split):
- MSCOCO: BiAug improved Image@1 Recall to 35.60% (vs. 34.59% for
CLIP-ftand 33.07% for base CLIP) and Text@1 to 55.46% (vs. 54.78% / 52.38%). - Flickr30K: Image@1 reached 65.36% (vs. 64.72% for
CLIP-ft), and Text@1 reached 82.20% (vs. 82.00%).
Limitations and Future Work
- Computational Overhead: Running LLM querying, open-vocabulary detection, and diffusion-based inpainting across hundreds of thousands of images is computationally expensive.
- Generative Imperfections: The foundation models used (GPT-3.5 and Stable Diffusion) still hallucinate or fail fine attribute generation (roughly ~37% inpainting inaccuracy in human breakdown tests).
- Word Order Blindness: While BiAug improved attribute and relation understanding, gains on purely syntactic word-order scrambling tests (Flickr30K-Order, COCO-Order) plateaued, indicating that attribute decoupling alone cannot fully resolve bag-of-words transformer behavior.
Summary updated