Abstract
Vision-language (VL) datasets often suffer from reporting bias, where humans omit obvious details in captions, leading models to struggle with basic object-attribute associations. We introduce BiAug, a bimodal augmentation framework that leverages Large Language Models (LLMs) and image inpainting to synthesize missing details and "hard negative" examples. By decoupling objects from their common attributes and re-pairing them diversely, BiAug significantly improves a model's understanding of physical world nuances, as demonstrated by superior performance on the ARO benchmark and standard retrieval tasks.
1. Introduction: The Challenge of Reporting Bias
When humans describe a scene, they naturally omit information considered "commonsense." For instance, a caption for a dog on a lawn might mention the dog's action but ignore the color of the grass or the dog's breed. This phenomenon, known as reporting bias, creates a gap for Vision-Language (VL) models like CLIP, which rely strictly on provided data. Without explicit mentions of "obvious" attributes, models fail to learn robust object-attribute associations and struggle to distinguish between specific visual nuances (e.g., "sliced salmon" vs. "whole fish").
2. Methodology: The BiAug Framework
To address this, we propose BiAug (Bimodal Augmentation), a pipeline designed to decouple objects from their stereotypical attributes and re-pair them with diverse, explicit descriptions.
2.1 Cross-Modal Object Extraction
The process begins by using an LLM to identify objects likely present in a scene based on its caption. A grounding object detector (GroundingDino) then verifies these objects in the image, uncovering elements present but unmentioned in the original text.
2.2 Decoupling Object–Attribute Association
For each detected object, the LLM generates detailed descriptions across categories such as color, shape, and material. Crucially, it also produces hard negatives—descriptions of the same object with contrasting attributes (e.g., changing "lush green grass" to "dry brown grass").
2.3 Bimodal Synthesis
To ensure consistency between the new text and the visual data, a Stable Diffusion inpainting model is employed. The original object is masked and regenerated according to the LLM’s detailed descriptions. This creates synthetic image-text pairs that explicitly label previously omitted details.
3. Experimental Results
BiAug was evaluated using subsets of the Conceptual Caption 3M (CC3M) dataset for training and several benchmarks for testing.
3.1 Compositionality and Attribution
On the ARO Benchmark, which tests Attribution, Relation, and Order, BiAug significantly outperformed standard CLIP fine-tuning. Notably, on the VG-Attribute subset, BiAug maintained steady performance gains as the training set size increased, whereas baseline models often showed performance degradation when exposed to more biased data.
3.2 General Retrieval Performance
BiAug also demonstrated consistent improvements in zero-shot image and text retrieval on standard datasets like MSCOCO and Flickr30K. On MSCOCO Image Retrieval (Recall@1), BiAug achieved a score of 35.60, surpassing the baseline of 34.59.
4. Conclusion
BiAug effectively mitigates reporting bias by using generative models to fill the "commonsense" gaps in human-annotated data. By creating bimodal hard negatives, the framework forces models to move beyond lazy pattern matching toward a more granular understanding of visual attributes. Future work will focus on reducing the computational overhead of the inpainting process and extending the framework to more complex relational scenarios.
Summary updated