Revisiting Visual Grounding

Erik Conser, Kennedy Hahn, Chandler Watson, Melanie MitchellNAACL 2019

TL;DR

This paper re-examines a popular visual grounding method, "Image Retrieval Using Scene Graphs" (IRSG), and finds that it doesn't significantly benefit from its relationship modeling capabilities. The authors also uncover exploitable biases in the commonly used IRSG dataset, suggesting that current evaluation metrics might not accurately reflect true model performance.

What Problem Does This Paper Address?

Visual grounding is the task of connecting elements of a structured description (like a scene graph or a subject-predicate-object triple) to corresponding objects and their relationships within an image. This is crucial for applications like automated image editing, captioning, and retrieval. While deep learning has made strides, visual grounding remains challenging due to the complexities of natural language parsing and the need for robust visual feature extraction, especially for attributes and spatial relationships. The authors specifically investigate the IRSG system, a prominent method in this area, to understand its actual performance and the validity of the datasets used to evaluate it. They question whether existing datasets truly test the intended capabilities of these models or if they contain biases that allow simpler methods to perform well.

Methods

The paper revisits the IRSG method, which converts natural language descriptions into scene graphs. It then uses this scene graph to find corresponding bounding boxes in an image, aiming to identify objects, their attributes, and their relationships. The IRSG system employs Convolutional Neural Networks (CNNs) for object and attribute appearance models and Gaussian Mixture Models (GMMs) for relationship models. These are integrated into a factor graph for inference.

To evaluate IRSG's reliance on relationship information, the authors introduce a simple object-only baseline. This baseline uses only object category probabilities derived from IRSG's trained CNNs, ignoring attributes and relationships. They compare the performance of IRSG and this baseline using a "recall at k" (R@k) metric, which measures how often the correct grounding appears within the top k ranked results.

Additionally, the authors perform a manual inspection of the IRSG dataset and its queries to identify potential biases. They also create a "clean dataset" by selecting queries that focus solely on objects and relationships (no attributes) and ensuring a higher number of positive examples per query, with corrected labels.

Datasets

  • IRSG Dataset: The primary dataset used for evaluation, adapted from Johnson et al. (2015). The authors note several issues and biases within this dataset.
  • Visual Relationship Dataset (VRD): Used in the context of revisiting "Referring Relationship" grounding, adapted from Krishna et al. (2018).
  • "Clean Dataset": A newly curated subset of the IRSG dataset, designed to mitigate biases by focusing on object-relationship queries without attributes and ensuring more positive examples per query.

Key Insights and Ideas

  • Relationships Don't Add Much: The IRSG system's performance on the original dataset is nearly identical to a simple object-only baseline. This suggests that the relationship modeling component of IRSG, despite being a core part of its design, does not significantly improve its ability to ground scene graphs.
  • Dataset Biases are Prevalent: The IRSG dataset (and by extension, the VRD dataset) contains exploitable biases. These include:
  • Uniqueness Bias: Positive test images often contain only one instance of each query object, making relationship information redundant.
  • Indistinguishability Bias: When multiple instances of query objects exist, any instance can serve as a correct grounding, again diminishing the need for precise relationship modeling.
  • Evaluation Metrics Matter: The authors propose a slightly modified R@k metric that they believe provides a more robust evaluation than the one used in the original IRSG paper.
  • "Clean" Data Still Shows Weak Relationship Performance: Even when using a curated "clean dataset" designed to remove biases, IRSG's performance remains very similar to the object-only baseline, reinforcing the idea that relationship information is not effectively utilized.

Key Results

  • On the partial scene graph queries from the original IRSG dataset, the IRSG model and the object-only baseline achieved nearly identical R@k performance curves (Figure 3).
  • A scatter plot analysis (Figure 4) showed a strong linear relationship between the factorization including relationships and the factorization without them, indicating that relationship probabilities contribute little to the final grounding energy. Over 90% of queries exhibited this strong linear relationship.
  • On the "clean dataset," the R@k curves for IRSG and the object-only baseline were again very similar (Figure 5).
  • For the "Referring Relationship" task, an object-only baseline (faster R-CNN) was able to correctly locate subjects and objects 56% of the time on the VRD dataset, indicating significant biases that question prior results.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that their study focused on partial scene graphs, which they considered more realistic for retrieval than complex full scene graphs.
  • The manual inspection for dataset biases was not exhaustive, with time constraints preventing a precise count of all biased instances.
  • The "clean dataset" was limited to seven queries involving only objects and relationships, excluding attributes.
  • Future Work:
    • Investigate other prominent visual grounding algorithms and datasets.
  • Curate benchmarks that are free from the identified biases and problems.
  • Explore alternative approaches to visual grounding, such as "active" grounding methods.
  • Replicate and compare Krishna et al.'s (2018) results with their object-only baseline on the VRD dataset.

Summary updated

BibTeX
@inproceedings{conser2019revisiting,
  author = {Erik Conser and Kennedy Hahn and Chandler Watson and Melanie Mitchell},
  title = {{Revisiting Visual Grounding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure