On the Role of Scene Graphs in Image Captioning

Dalin Wang, Daniel Beck, Trevor CohnLANTERN @ EMNLP 2019 · OpenReview

Abstract This paper investigates whether the inherent noise in automatically generated scene graphs limits their utility in image captioning. By comparing state-of-the-art parsers against human-annotated ground-truth data, we demonstrate that current captioning models are the primary bottleneck. Our results show that models perform nearly as well with noisy, automatically generated graphs as they do with perfect, human-annotated ones, provided that appropriate gating mechanisms are employed.

1. Introduction

Deep learning has significantly advanced image captioning, yet generated descriptions often lack semantic depth, frequently identifying objects without capturing their nuanced relationships. Scene Graphs (SGs)—structured representations where nodes denote objects and edges denote relationships—offer a promising avenue for injecting this semantic detail. However, the reliance on "ad-hoc" parsers introduces significant noise. This study addresses a fundamental question: is the inaccuracy of scene graph parsers the limiting factor, or does the fault lie in how captioning models process this structured information?

2. Methodology: Integrating Scene Graphs and Visual Features

We propose a captioning architecture that extends the "Bottom-Up and Top-Down" (BUTD) framework to incorporate graph-based semantic data.

2.1 Scene Graph Encoder

The model utilizes a Graph Convolutional Network (GCN) to process the scene graph. A critical innovation is the implementation of edge-wise gating. This mechanism allows the model to mathematically "ignore" or prune edges that are deemed noisy or irrelevant during training, effectively filtering the parser's output.

2.2 Cascaded Attention

To integrate visual and semantic features, we employ a two-step cascaded attention mechanism. The model first attends to features extracted from the scene graph via the GCN. This semantic context then guides a second attention step over specific regions in the original image, ensuring that the generated text is grounded in both structured relationships and raw visual data.

The proposed architecture incorporating scene graphs and image regions via cascaded attention.

3. Experimental Setup

Experiments were conducted on the MS-COCO dataset, using Visual Genome for ground-truth scene graphs. To isolate the impact of parser noise, we created the MS-COCO-GT subset, where images have both MS-COCO captions and Visual Genome annotations. This allows for a direct "Oracle" comparison between predicted graphs (generated by Factorizable-Net) and perfect, human-annotated graphs.

4. Results and Analysis

Our findings suggest that the parser is not the primary bottleneck in current systems.

4.1 Quantitative Performance

On the full MS-COCO dataset, the addition of scene graphs with edge-wise gating improved the CIDEr score from 108.0 to 108.6 and the SPICE score from 19.9 to 20.3. The improvement in SPICE, which measures semantic propositional content, confirms that graphs enhance the model's understanding of "what" is happening in the scene.

In the MS-COCO-GT "Oracle" experiments, the model using predicted graphs achieved a CIDEr of 103.4, while the model using ground-truth graphs achieved 105.2. This narrow 1.8-point gap indicates that even imperfect scene graphs provide most of the semantic value required by current architectures.

4.2 Qualitative Analysis

Qualitative results demonstrate that the combined "Image + Graph" (I + G) model is superior at identifying spatial relationships. For instance, while image-only models might miss background details, the I + G model correctly identifies complex interactions, such as a man riding a motorcycle "in front of a white bus."

Qualitative comparison of captions generated using image features, scene graphs, and their combination.

5. Discussion and Conclusion

The study reveals that the primary challenge in leveraging scene graphs lies in the model architecture rather than parser accuracy. While edge-wise gating is effective at mitigating noise, the small performance gain from ground-truth graphs suggests that current LSTM-based decoders may not be fully exploiting the structured information provided.

Future work should explore expanding the vocabulary of scene graph parsers beyond the current 150 object and 50 relationship categories. Additionally, moving toward end-to-end multi-task frameworks—where the system learns to predict scene graphs and captions simultaneously—could further unlock the potential of structured semantic representations in vision-to-language tasks.

Summary updated

BibTeX
@inproceedings{wang2019role,
  author = {Dalin Wang and Daniel Beck and Trevor Cohn},
  title = {{On the Role of Scene Graphs in Image Captioning}},
  booktitle = {LANTERN @ EMNLP},
  year = {2019},
  url = {https://openreview.net/forum?id=VCJD___scWB},
}
Figure