Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association

Qiyu Wu, Mengjie Zhao, Yutong He, Lang Huang, Junya Ono, Hiromi Wakaki, Yuki MitsufujiSubmission to ICLR 2024 · Rejected

TL;DR

Vision-language datasets suffer from reporting bias: people rarely describe obvious commonsense details (like grass being green or a dog being brown), causing models to latch onto spurious object–attribute correlations. The authors introduce BiAug, a pipeline that uses large language models (LLMs) and diffusion inpainting to decouple objects from their default attributes, synthesizing aligned image–text pairs and cross-modal hard negatives. Fine-tuning CLIP with BiAug boosts compositionality and zero-shot image-text retrieval without requiring new human annotations.

What Problem Does This Paper Address?

When humans write captions, they tend to omit details that feel like common sense. A caption might say "a dog runs with a tennis ball," omitting whether the dog is brown or black, or whether it runs on snow or grass. This tendency is known as reporting bias.

For multimodal foundation models like CLIP, reporting bias is toxic:

  1. Ambiguous pairings: The exact same caption can match visually disparate images.
  2. Spurious feature binding: Models overfit to dominant co-occurrences (e.g., assuming a "lawn" is always green, or associating "flag" almost exclusively with the American flag), failing to decouple objects from their attributes.
  3. Bag-of-words behavior: Vision-language models often struggle to tell the difference between "a white road and a paved house" vs. "a paved road and a white house."

Existing dataset-cleaning works focus either on caption bootstrapping or image-only generation, but neither directly fixes the joint image–text reporting bias.

Methods

To break the tight coupling between objects and their frequent attributes, BiAug (Bimodal Augmentation) applies a three-phase pipeline:

Source Pair ──► [Phase I: LLM + GroundingDINO] ──► Object Detection & Visual Filtering
                      │
                      ▼
                [Phase II: LLM Decoupling]      ──► Generate Diverse & Hard Negative Attributes
                      │
                      ▼
                [Phase III: Stable Diffusion]    ──► Mask & Inpaint Contrasting Attributes
                      │
                      ▼
                Augmented Bimodal Hard-Negative Pairs
  1. LLM-Aided Object Detection: Given an initial caption, an LLM (gpt-3.5-turbo) infers up to five objects likely present in the scene—especially unmentioned background objects. GroundingDINO then grounds and verifies these candidates on the image, filtering out false positives and low-confidence detections.
  2. Object–Attribute Decoupling via LLM: For each validated object, the LLM generates descriptive attributes spanning four categories (color, shape, material, and other). Crucially, it also generates an opposing "hard negative" attribute (e.g., “lush green grass” vs. “dry grass”), inserting both into the original caption to produce paired contrasting captions.
  3. Image Inpainting & Negative Construction: Using bounding boxes from Phase 1, the object area is masked. A pre-trained diffusion inpainter (Stable Diffusion Inpainting) modifies the region according to the contrasting descriptions.

This creates balanced, bimodal hard-negative triplets where the text and the visual appearance vary strictly across specific attribute dimensions. The model (a standard CLIP ViT-B/16) is fine-tuned using contrastive ranking loss with these samples treated as in-batch negatives.

Datasets

  • Training/Augmentation: Subsets of Conceptual Captions 3M (CC3M) scaled at 40K, 100K, 200K, and 300K samples. CC3M was selected because its captions naturally use hypernyms with well-defined attribute spaces.
  • Compositionality Evaluation: The ARO (Attribution, Relation, and Order) benchmark (~50k test cases), focusing on:
  • VG-Attribute and VG-Relation (swapped objects and attributes).
  • Flickr30K-Order and COCO-Order (word/phrase order scrambling).
  • Standard Retrieval Evaluation: Karpathy test splits of MSCOCO and Flickr30K.

Key Insights and Ideas

  • Omission is widespread: In human evaluation, ~60% of the objects proposed by the LLM and grounded by the detector were completely absent from the original caption, confirming that text-image pairs systematically omit visual context.
  • BiAug stabilizes scaling: When fine-tuning vanilla CLIP (CLIP-ft) on larger amounts of raw CC3M data, performance on compositional benchmarks (ARO) actually degraded due to reinforcement of biased shortcuts. In contrast, BiAug continuously improved as more augmented data was added.
  • Filtering is non-negotiable: Inpainting tools are imperfect (human evaluators judged inpainting attribute correctness at ~63%). Bounding-box thresholding (rejecting boxes under 5% or over 80% of image size, or with >70% overlap) and confidence filtering are essential; removing filtering caused severe performance drops.

Key Results

  • Compositionality (ARO Benchmark, 300K subset):
  • VG-Relation Macro Accuracy: BiAug reached ~69.3%, outperforming standard fine-tuning (~67.0%) and zero-shot CLIP (~63.7%).
  • VG-Attribute Macro Accuracy: BiAug achieved ~64.0%, whereas standard fine-tuning plateaued and dropped to ~62.4%.
  • Cross-Modal Retrieval (Karpathy Split):
  • MSCOCO: BiAug improved Image@1 Recall to 35.60% (vs. 34.59% for CLIP-ft and 33.07% for base CLIP) and Text@1 to 55.46% (vs. 54.78% / 52.38%).
  • Flickr30K: Image@1 reached 65.36% (vs. 64.72% for CLIP-ft), and Text@1 reached 82.20% (vs. 82.00%).

Limitations and Future Work

  • Computational Overhead: Running LLM querying, open-vocabulary detection, and diffusion-based inpainting across hundreds of thousands of images is computationally expensive.
  • Generative Imperfections: The foundation models used (GPT-3.5 and Stable Diffusion) still hallucinate or fail fine attribute generation (roughly ~37% inpainting inaccuracy in human breakdown tests).
  • Word Order Blindness: While BiAug improved attribute and relation understanding, gains on purely syntactic word-order scrambling tests (Flickr30K-Order, COCO-Order) plateaued, indicating that attribute decoupling alone cannot fully resolve bag-of-words transformer behavior.

Summary updated

Abstract

Reporting bias arises when people assume that some knowledge is universally understood and hence, do not necessitate explicit elaboration. In this paper, we focus on the wide existence of reporting bias in visual-language datasets, embodied as the object-attribute association, which can subsequentially degrade models trained on them. To mitigate this bias, we propose a bimodal augmentation (BiAug) approach through object-attribute decoupling to flexibly synthesize visual-language examples with a rich array of object-attribute pairing and construct cross-modal hard negatives. We employ large language models (LLMs) in conjunction with a grounding object detector to extract target objects. Subsequently, the LLM generates a detailed attribute description for each object and produces a corresponding hard negative counterpart. An inpainting model is then used to create images based on these detailed object descriptions. By doing so, the synthesized examples explicitly complement omitted objects and attributes to learn, and the hard negative pairs steer the model to distinguish object attributes. Our experiments demonstrated that BiAug is superior in object-attribute understanding. In addition, BiAug also improves the performance on zero-shot retrieval tasks on general benchmarks like MSCOCO and Flickr30K. BiAug refines the way of collecting text-image datasets. Mitigating the reporting bias helps models achieve a deeper understanding of visual-language phenomena, expanding beyond mere frequent patterns to encompass the richness and diversity of real-world scenarios.
BibTeX
@misc{wu2023towards,
  author = {Qiyu Wu and Mengjie Zhao and Yutong He and Lang Huang and Junya Ono and Hiromi Wakaki and Yuki Mitsufuji},
  title = {{Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association}},
  year = {2023},
  eprint = {2310.01330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01330},
}
Figure