Flexible Visual Grounding

Yongmin Kim, Chenhui Chu, Sadao KurohashiACL 2022

TL;DR

This paper introduces "flexible visual grounding," a new task that addresses the real-world challenge of queries that cannot be grounded to an image. They propose a novel method to handle these unanswerable queries by introducing a "pseudo region" and create two new datasets to train and evaluate their model, achieving high accuracy on both answerable and unanswerable cases.

What Problem Does This Paper Address?

Visual grounding, a task that aims to find a specific region in an image corresponding to a textual query (like "the red car"), is crucial for many vision-language tasks. However, existing visual grounding research and datasets assume that every query can be grounded to an image. This assumption breaks down in real-world scenarios like social media, where text and images aren't always directly related, leading to many "unanswerable" queries. This limitation hinders the development of robust vision-language models that can handle the messiness of real-world data. This paper tackles this gap by introducing the problem of flexible visual grounding, which accommodates both answerable and unanswerable queries.

Methods

The core innovation is a method to handle unanswerable queries. Instead of simply failing, their model is trained to assign a "pseudo region" to unanswerable queries. This pseudo region acts as a placeholder, allowing the model to learn to distinguish between queries that can be grounded and those that cannot.

The overall model architecture builds upon existing visual grounding approaches, specifically using a Faster R-CNN for region proposal and a multi-task ViLBERT model for calculating similarity scores between region proposals and the input query. The training objective is a binary cross-entropy loss, modified to accommodate the pseudo region for unanswerable queries.

Datasets

The authors introduce two new datasets to study flexible visual grounding:

  • RefCOCO+ Pseudo Dataset: This dataset is constructed by taking the existing RefCOCO+ dataset and creating unanswerable query-image pairs by randomly pairing queries from one image with another. This introduces noise but provides a large-scale pseudo-dataset.
  • SMD4FVG (Social Media Dataset for Flexible Visual Grounding): This dataset is built from Twitter data. It involves a multi-stage filtering process (image classification, object detection, text detection) to select relevant tweets, followed by query extraction and crowdsourced annotation to identify both answerable and unanswerable queries. This dataset aims to capture more realistic unanswerable queries.

Key Insights and Ideas

  • Real-world data is messy: The fundamental assumption that all queries can be grounded is a significant limitation for current visual grounding models.
  • Pseudo regions are effective: Introducing a "pseudo region" for unanswerable queries is a simple yet effective way to train a model to handle cases where a query doesn't correspond to any visual element.
  • Social media is a rich source of unanswerable queries: The SMD4FVG dataset highlights the prevalence and complexity of unanswerable queries in real-world social media content.
  • Unanswerable queries are easier to learn than answerable ones: The authors observe that their model generally achieves higher accuracy on unanswerable queries than on answerable ones, suggesting that distinguishing "not there" is often easier than precisely locating "there."

Key Results

  • On the RefCOCO+ Pseudo Dataset, their model achieved 91.2% accuracy on unanswerable queries and 69.7% on answerable queries, demonstrating its ability to handle both.
  • On the SMD4FVG Dataset, the "SM" setting (training directly on SMD4FVG) achieved the best overall accuracy of 81.7%, with a remarkable 95.0% accuracy on unanswerable queries.
  • The "Pseudo→SM" setting, which fine-tuned a model pre-trained on the pseudo dataset with the SMD4FVG dataset, achieved a good trade-off, improving answerable query performance over the "SM" setting while maintaining high unanswerable query accuracy.

Limitations and Future Work

  • Pseudo dataset noise: The RefCOCO+ pseudo dataset, created by random pairing, may contain some noise where randomly paired queries are coincidentally answerable.
  • Dataset constraints: The SMD4FVG dataset was constrained to RefCOCO+ object classes, which might limit the model's generalization to real-world data with a broader range of objects.
  • Ambiguous queries: The paper acknowledges that some queries are inherently ambiguous, making it difficult to definitively label them as answerable or unanswerable, and that their model sometimes makes mistakes on these.
  • Future work: The authors plan to construct a dataset without the RefCOCO+ class constraints to better evaluate real-world performance and improve handling of ambiguous queries.

Summary updated

BibTeX
@inproceedings{kim2022flexible,
  author = {Yongmin Kim and Chenhui Chu and Sadao Kurohashi},
  title = {{Flexible Visual Grounding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure