VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training.

Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, Zicheng Liu

Introduction

Standard image captioning models are often limited by a "vocabulary gap." While they excel at describing common objects found in datasets like COCO (which contains roughly 80 categories), they struggle when encountering "novel" objects in the wild—such as an accordion or a dolphin—because they have never seen these words used in a descriptive sentence. The nocaps (novel object captioning) challenge was established to address this, forbidding the use of additional image-caption pairs for training. This necessitates a method that can learn to describe thousands of objects using only image-level tags and a limited set of full captions.

The VIVO Framework

To bridge this gap, the authors introduced VIVO (VIsual VOcabulary), a novel pre-training framework that decouples object recognition from linguistic composition. The training process consists of two distinct stages:

1. VIVO Pre-training

In the first stage, the model learns a "visual vocabulary" by aligning image regions with simple object tags rather than full captions. Using the Open Images V5 dataset, which contains 1.7 million images across 6.4k unique object classes, the model is trained on image-tag pairs.

The VIVO pre-training framework aligns image regions with tags to build a visual vocabulary.

The architecture utilizes a multi-layer Transformer and employs two key techniques:

  • Masked Tag Prediction: Similar to BERT, the model masks certain tags and attempts to predict them based on image regions and the surrounding tags.
  • Hungarian Matching Loss: Since tags are provided as an unordered "bag of words," this loss function finds the optimal one-to-one mapping between predictions and ground-truth tags, ensuring the model learns robust visual-semantic alignments without relying on sequence order.

2. Fine-tuning

After pre-training, the model is fine-tuned on the COCO dataset. During this stage, it learns the "grammar" and structure of natural language sentences. Crucially, the model retains the vast visual vocabulary acquired during pre-training, allowing it to generalize its linguistic knowledge to the thousands of objects it can now recognize.

Key Insights: Semantic Proximity and Grounding

A core strength of VIVO is its ability to learn a joint embedding space where semantically similar objects are mapped to nearby vectors. For instance, if the model learns how to use the word "instrument" in a sentence during fine-tuning, it can easily substitute "accordion" because the two are close in the embedding space. This semantic proximity allows for zero-shot generalization to novel objects. Furthermore, the pre-training stage ensures precise visual grounding, forcing the model to align specific image regions with their corresponding tags, rather than hallucinating labels.

Experimental Results

VIVO achieved state-of-the-art performance on the nocaps benchmark, which evaluates models on nearly 400 objects rarely or never seen in COCO.

VIVO successfully captions a novel object (accordion) by leveraging its pre-trained visual vocabulary.

Key results include:

  • Surpassing Human Performance: VIVO achieved a CIDEr score of 88.3 on the overall nocaps dataset, outperforming the human baseline of 87.1.
  • Out-of-Domain Generalization: In the most challenging "out-of-domain" category, VIVO reached an F1-score of 30.6%, nearly doubling the performance of previous state-of-the-art models like OSCAR (15.7%).
  • Efficiency: Unlike many competitors, a single VIVO model outperformed complex ensembles and methods using post-processing tricks like Constrained Beam Search.

Conclusion and Future Directions

VIVO demonstrates that large-scale image-tag pre-training is a powerful alternative to traditional image-caption pre-training for expanding a model's visual vocabulary. While the system currently depends on the quality of the object detector and the diversity of the training tags, future work may involve scaling VIVO using machine-generated tags from the web. This could potentially enable models to learn an almost unlimited visual vocabulary without the need for extensive human labeling.

Summary updated

Abstract

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other than COCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.
BibTeX
@inproceedings{hu2021vivo,
  author = {Xiaowei Hu and Xi Yin and Kevin Lin and Lijuan Wang and Lei Zhang and Jianfeng Gao and Zicheng Liu},
  title = {{VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16249},
}
Figure