Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering

Soravit Changpinyo, Bo Pang, Piyush Sharma, Radu Soricut

Vision-language models, which power tasks like image captioning and visual question answering (VQA), have traditionally relied on integrated object detectors. However, these detectors often suffer from a "vocabulary gap" due to the high cost of annotating bounding boxes with specific labels. Research from Google AI proposes a novel solution: decoupling the act of finding an object (box proposal) from the act of describing it (featurization). By leveraging web-scale data and "ultrafine-grained" semantic labels, this approach significantly improves model performance on diverse, real-world imagery.

The Bottleneck of Integrated Detection

Most contemporary AI systems use "bottom-up" signals from detectors like Faster R-CNN, typically trained on datasets such as Visual Genome. These datasets require human annotators to provide both precise bounding boxes and category labels. This creates two primary limitations:

  1. Scalability: The dual requirement of boxes and labels makes annotation expensive, limiting dataset size.
  2. Granularity: Due to limited training categories, detectors often default to coarse labels. For instance, a model might identify "money" rather than a specific denomination, or a generic "bridge" instead of a named landmark.

This lack of detail hinders downstream tasks. A VQA model cannot answer "How much money is this?" if it only perceives the object as generic currency. Furthermore, these models often struggle with "out-of-domain" images—unstructured, real-world photos that differ from curated training sets.

A Decoupled Architecture

The researchers propose separating the detection pipeline into two independent components:

  • The Box Proposer: A standard Faster R-CNN trained on Visual Genome acts as a category-agnostic proposer. Its sole responsibility is identifying regions likely to contain objects.
  • The "Ultra" Featurizer: Instead of using internal detector features, the model crops the proposed regions and processes them through Graph-RISE (Graph-Regularized Image Semantic Embedding).

Graph-RISE is the core innovation. Trained on 260 million web images with 40 million noisy, image-level labels, it does not require bounding box annotations for training. This allows it to learn an ultrafine-grained semantic space capable of distinguishing between specific landmarks, food types, and currency denominations.

The final architecture integrates three feature types: Global (whole image), Box (object crops via Graph-RISE), and Label (semantic embeddings from the Google Cloud Vision API).

Evaluation on Real-World Datasets

The approach was tested on two challenging datasets that reflect real-world complexity:

  • Conceptual Captions (CC): Containing 3.3 million web-scraped images and alt-text, this dataset is significantly more diverse and "messy" than standard benchmarks.
  • VizWiz: A VQA dataset of photos taken by visually impaired individuals. These images are often poorly framed or blurry and contain highly specific objects, making them an ideal test for transfer learning.

Key Results

The "Ultra" features consistently outperformed standard object detection baselines:

  • Image Captioning: On the Conceptual Captions leaderboard, the decoupled model (G + B-Ultra + L) achieved a CIDEr score of 0.984, compared to the baseline's 0.944.
  • Visual Question Answering: On VizWiz, the model reached 53.7% accuracy (a 1.8% absolute improvement). Notably, it saw significant gains in "number" questions (+4.5%) and open-ended "other" questions (+3.3%), where granularity is critical.

Qualitative Successes

The strength of ultrafine-grained labels is most apparent in qualitative comparisons. The decoupled model demonstrates a superior ability to identify specific entities that generic detectors miss.

Qualitative win: The decoupled model correctly identifies 'monks' where the baseline model predicted 'woman'.
The 'Ultra' featurizer identifies specific items like 'chia seeds' instead of generic 'dried flowers'.

Conclusion and Future Directions

By decoupling "where" an object is from "what" it is, researchers can utilize massive, image-level datasets to enhance object-level understanding. This approach provides the "freedom to annotate" at scale, leading to models that are more robust to unfamiliar objects and capable of generating more precise descriptions.

While the current model relies on a fixed box proposer, future work may explore truly category-agnostic proposers or higher-dimensional embeddings to capture even more nuanced semantic details. This research marks a significant step toward vision-language models that can truly "see" the world in all its granular detail.

Summary updated

Abstract

Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering. However, popular models like Faster R-CNN rely on a costly process of annotating ground-truths for both the bounding boxes and their corresponding semantic labels, making it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and featurization for down-stream tasks. The key insight is that this allows us to leverage a large amount of labeled annotations that were previously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and improved image captioning and visual question answering models, as measured on publicly available benchmarks.
BibTeX
@inproceedings{changpinyo2019decoupled,
  author = {Soravit Changpinyo and Bo Pang and Piyush Sharma and Radu Soricut},
  title = {{Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=moMV_gJ3hMd},
}
Figure