Large-Scale Bidirectional Training for Zero-Shot Image Captioning

Taehoon Kim, Mark Marsden, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Alessandra Sala, Seung Hwan KimWorkshops @ CVPR 2024 · DOI · arXiv

Introduction

Image captioning models typically follow a "pretrain-then-finetune" pipeline, which often limits their ability to generalize to new domains without additional training. The BITTERS framework addresses this by employing large-scale bidirectional training on a curated dataset of 100 million image-text pairs. This approach allows the model to generate high-quality, diverse captions in a zero-shot manner while maintaining computational efficiency.

Methodology

The BITTERS framework introduces several technical innovations to improve both performance and efficiency.

WaveVAE: Wavelet-Based Image Reconstruction

A core component of BITTERS is WaveVAE, which utilizes a 2D Discrete Wavelet Transform (DWT) instead of standard downsampling. By decomposing images into high-pass and low-pass frequencies, WaveVAE achieves high-fidelity reconstruction with 75% fewer parameters than previous hierarchical VAE architectures. This efficiency is vital for scaling the model to massive datasets.

BiART: Bidirectional Training

The model uses a Bidirectional Auto-Regressive Transformer (BiART), trained to predict text from images and images from text simultaneously. This bidirectional objective acts as a powerful regularizer, stabilizing the learning of the joint distribution between modalities and preventing model divergence during large-scale training.

A sample image used to test zero-shot captioning accuracy.

Parameter-Efficient Adaptation

To enable specific tasks like keyword extraction without compromising zero-shot capabilities, the authors use adapters. These small modules add only 0.02% more parameters, allowing the model to learn new tasks while keeping the core pre-trained weights frozen.

Datasets and Evaluation

The authors emphasize that data quality is a primary driver of zero-shot performance. BITTERS is trained on TIP100M, a curated set of 100 million professionally moderated image-text pairs from Shutterstock. Evaluation is conducted using two new benchmarks:

  • ICE-A (Accuracy): 5,100 images across 17 categories to test zero-shot generalization.
  • ICE-B (Bias): 22,000 images with demographic metadata to measure societal bias.

Key Results

BITTERS significantly outperforms previous state-of-the-art zero-shot models. On the MS-COCO benchmark, it achieved a CIDEr score of 31.2, more than doubling the performance of ZeroCap (14.6). In terms of efficiency, BITTERS uses 32% less GPU memory and is 18% faster at sampling compared to previous leaders like L-Verse.

Qualitative results of BITTERS zero-shot image captioning across various categories.

In human preference studies, BITTERS-generated captions were preferred 34.9% of the time over competitors. However, the authors also noted that increased accuracy often correlates with higher societal bias scores, highlighting a trade-off that requires further research into training set auditing and bias mitigation.

Conclusion

BITTERS demonstrates that high-quality zero-shot image captioning can be achieved through efficient architectural choices and a focus on data curation. By combining wavelet-based VAEs with bidirectional training, the framework provides a scalable and effective solution for generating descriptive and accurate image captions.

Summary updated

BibTeX
@inproceedings{kim2024large,
  author = {Taehoon Kim and Mark Marsden and Pyunghwan Ahn and Sangyun Kim and Sihaeng Lee and Alessandra Sala and Seung Hwan Kim},
  title = {{Large-Scale Bidirectional Training for Zero-Shot Image Captioning}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00733},
}
Figure