TL;DR
This paper investigates whether scene graphs, which explicitly represent objects and their relationships in an image, can improve image captioning. They introduce a novel conditional graph attention mechanism (C-GAT) but find that current scene graph generation models are too noisy to consistently benefit captioning. However, when high-quality scene graphs are used, significant improvements in captioning quality are observed.
What Problem Does This Paper Address?
Generating descriptive captions for images is a challenging task in computer vision. While state-of-the-art image captioning models primarily rely on object detection features, there's a growing interest in incorporating relational information between objects. Scene graphs offer a structured way to represent these relationships. However, it's unclear if the current quality of automatically generated scene graphs is sufficient to provide a tangible benefit to image captioning, or if the noise in these graphs actually hinders performance. This paper aims to empirically answer whether scene graphs, and specifically how their quality, impact image captioning.
Methods
The authors propose a Conditional Graph Attention Network (C-GAT). This is an extension of standard Graph Attention Networks (GATs) that allows the graph updates to be conditioned on external context, specifically the current state of the image captioning decoder. This aims to make the scene graph features more relevant to the ongoing caption generation process.
They experiment with two main attention mechanisms for integrating scene graph features with object features into an LSTM decoder:
- Flat Attention (FA): Two independent attention heads process object features and scene graph features separately, and their outputs are combined.
- Hierarchical Attention (HA): The output of one attention head is used as input to the other, creating a sequential processing of information. They explore two variants: HA-SG (scene graph features first) and HA-IM (object features first).
They also compare their proposed C-GAT approach with a standard GAT layer for processing scene graphs.
Datasets
The primary dataset used for training and evaluation is MSCOCO. They use the standard Karpathy split, with 113k images for training and 5k for validation/testing.
For a more in-depth analysis of scene graph quality, they use a subset of MSCOCO that overlaps with Visual Genome, referred to as VG-COCO. This dataset allows them to evaluate models on images with both captions and scene graph annotations.
Key Insights and Ideas
- Noise is the Enemy: The biggest takeaway is that the quality of the predicted scene graph is paramount. Current scene graph generation models are too noisy, and this noise can negatively impact captioning performance, even leading to worse results than models that don't use scene graphs at all.
- Context Matters: Their proposed C-GAT mechanism, which conditions graph updates on the decoder state, shows some promise, suggesting that making scene graph processing context-aware is beneficial.
- Hierarchical Attention is Better (Sometimes): The hierarchical attention mechanism, particularly when object features are processed first (HA-IM), tends to perform better than flat attention, especially when scene graph quality is low. This suggests that using more reliable object features to guide the attention over potentially noisy scene graph features is a good strategy.
- Relations Gain, Objects Lose: When scene graphs are used, there's a consistent small gain in the ability to describe relations (measured by SPICE relation F1 scores), but this often comes at the cost of describing objects, leading to lower overall SPICE scores.
Key Results
- No Consistent Gain with Noisy Graphs: Across various metrics (BLEU-4, CIDEr, ROUGE-L, SPICE), models that incorporate scene graph features from standard SGG models do not consistently outperform the strong Bottom-Up Top-Down (BUTD) baseline. In some cases, they perform worse.
- C-GAT Shows Modest Improvement: The C-GAT architecture, when applied to scene graphs, leads to a small improvement of up to 0.8 CIDEr compared to a standard GAT.
- High-Quality Graphs Yield Significant Gains: When using high-quality scene graphs (e.g., from the top 33% of predicted graphs in VG-COCO), models can achieve substantial improvements, with gains of up to 3.3 CIDEr compared to the BUTD baseline.
- SPICE Breakdown: Using scene graphs leads to a small increase in relation F1 scores but a decrease in object F1 scores, resulting in worse overall SPICE scores when noise is present.
Limitations and Future Work
- Scene Graph Generation Quality: The primary limitation identified is the noise and low quality of current scene graph generation models. This significantly hinders their usefulness in downstream tasks like image captioning.
- Mismatch in Ground-Truth vs. Predicted: Experiments with ground-truth scene graphs show that models trained on predicted graphs perform worse when evaluated on ground-truth graphs, highlighting a potential mismatch in learned features and quality.
- Future Work: The authors suggest further research into improving scene graph generation models to reduce noise. They also propose exploring the application of their C-GAT mechanism to other tasks like visual question-answering and syntax-aware neural machine translation. Additionally, they plan to investigate how to selectively use predicted scene graphs based on their quality and how to account for highly detailed objects/relations in captioning.
Summary updated