Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association

Qiyu Wu, Mengjie Zhao, Yutong He, Lang Huang, Junya Ono, Hiromi Wakaki, Yuki Mitsufuji

Abstract

Vision-language (VL) datasets often suffer from reporting bias, where humans omit obvious details in captions, leading models to struggle with basic object-attribute associations. We introduce BiAug, a bimodal augmentation framework that leverages Large Language Models (LLMs) and image inpainting to synthesize missing details and "hard negative" examples. By decoupling objects from their common attributes and re-pairing them diversely, BiAug significantly improves a model's understanding of physical world nuances, as demonstrated by superior performance on the ARO benchmark and standard retrieval tasks.

1. Introduction: The Challenge of Reporting Bias

When humans describe a scene, they naturally omit information considered "commonsense." For instance, a caption for a dog on a lawn might mention the dog's action but ignore the color of the grass or the dog's breed. This phenomenon, known as reporting bias, creates a gap for Vision-Language (VL) models like CLIP, which rely strictly on provided data. Without explicit mentions of "obvious" attributes, models fail to learn robust object-attribute associations and struggle to distinguish between specific visual nuances (e.g., "sliced salmon" vs. "whole fish").

Illustration of reporting bias in standard datasets and the BiAug approach to generating bimodal hard negatives.

2. Methodology: The BiAug Framework

To address this, we propose BiAug (Bimodal Augmentation), a pipeline designed to decouple objects from their stereotypical attributes and re-pair them with diverse, explicit descriptions.

2.1 Cross-Modal Object Extraction

The process begins by using an LLM to identify objects likely present in a scene based on its caption. A grounding object detector (GroundingDino) then verifies these objects in the image, uncovering elements present but unmentioned in the original text.

2.2 Decoupling Object–Attribute Association

For each detected object, the LLM generates detailed descriptions across categories such as color, shape, and material. Crucially, it also produces hard negatives—descriptions of the same object with contrasting attributes (e.g., changing "lush green grass" to "dry brown grass").

2.3 Bimodal Synthesis

To ensure consistency between the new text and the visual data, a Stable Diffusion inpainting model is employed. The original object is masked and regenerated according to the LLM’s detailed descriptions. This creates synthetic image-text pairs that explicitly label previously omitted details.

The BiAug pipeline, showing the transition from original examples to augmented bimodal pairs via object detection, LLM-based attribute generation, and image inpainting.

3. Experimental Results

BiAug was evaluated using subsets of the Conceptual Caption 3M (CC3M) dataset for training and several benchmarks for testing.

3.1 Compositionality and Attribution

On the ARO Benchmark, which tests Attribution, Relation, and Order, BiAug significantly outperformed standard CLIP fine-tuning. Notably, on the VG-Attribute subset, BiAug maintained steady performance gains as the training set size increased, whereas baseline models often showed performance degradation when exposed to more biased data.

3.2 General Retrieval Performance

BiAug also demonstrated consistent improvements in zero-shot image and text retrieval on standard datasets like MSCOCO and Flickr30K. On MSCOCO Image Retrieval (Recall@1), BiAug achieved a score of 35.60, surpassing the baseline of 34.59.

4. Conclusion

BiAug effectively mitigates reporting bias by using generative models to fill the "commonsense" gaps in human-annotated data. By creating bimodal hard negatives, the framework forces models to move beyond lazy pattern matching toward a more granular understanding of visual attributes. Future work will focus on reducing the computational overhead of the inpainting process and extending the framework to more complex relational scenarios.

Summary updated

Abstract

Reporting bias arises when people assume that some knowledge is universally understood and hence, do not necessitate explicit elaboration. In this paper, we focus on the wide existence of reporting bias in visual-language datasets, embodied as the object-attribute association, which can subsequentially degrade models trained on them. To mitigate this bias, we propose a bimodal augmentation (BiAug) approach through object-attribute decoupling to flexibly synthesize visual-language examples with a rich array of object-attribute pairing and construct cross-modal hard negatives. We employ large language models (LLMs) in conjunction with a grounding object detector to extract target objects. Subsequently, the LLM generates a detailed attribute description for each object and produces a corresponding hard negative counterpart. An inpainting model is then used to create images based on these detailed object descriptions. By doing so, the synthesized examples explicitly complement omitted objects and attributes to learn, and the hard negative pairs steer the model to distinguish object attributes. Our experiments demonstrated that BiAug is superior in object-attribute understanding. In addition, BiAug also improves the performance on zero-shot retrieval tasks on general benchmarks like MSCOCO and Flickr30K. BiAug refines the way of collecting text-image datasets. Mitigating the reporting bias helps models achieve a deeper understanding of visual-language phenomena, expanding beyond mere frequent patterns to encompass the richness and diversity of real-world scenarios.
BibTeX
@misc{wu2023towards,
  author = {Qiyu Wu and Mengjie Zhao and Yutong He and Lang Huang and Junya Ono and Hiromi Wakaki and Yuki Mitsufuji},
  title = {{Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association}},
  year = {2023},
  eprint = {2310.01330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01330},
}
Figure