Leveraging Partial Dependency Trees to Control Image Captions

Wenjie Zhong, Yusuke MiyaoNAACL 2021

TL;DR

This paper introduces a novel method for controlling image caption generation by using partial syntactic dependency trees as explicit control signals. Their proposed Syntactic Dependency Structure Aware Model (SDSAM) learns to generate captions that not only include specified words but also adhere to a desired syntactic structure, leading to more precise and controllable image descriptions.

What Problem Does This Paper Address?

Controlling the output of image captioning models is a growing area of research, with existing methods focusing on attributes like style, sentiment, or specific objects. However, a significant gap remains in controlling the syntactic structure of the generated captions. While models can be guided to include certain words, the way these words are grammatically arranged can still vary widely, leading to ambiguity or less natural-sounding captions. For instance, specifying "dog, man, frisbee" might result in captions like "a dog and a man play frisbee on grass" or "a dog playing with a man catches frisbee," both using the same words but with different grammatical constructions. This paper aims to bridge this gap by enabling fine-grained control over both the words and their syntactic relationships within generated captions.

Methods

The core of this work is the Syntactic Dependency Structure Aware Model (SDSAM), a pipeline that leverages partial dependency trees for control.

  1. Partial Dependency Trees as Control Signals: The authors define a partial dependency tree as a sub-tree of a full syntactic dependency tree. This structure explicitly encodes words and their grammatical relationships (e.g., subject-verb, verb-object), making it a powerful signal for controlling caption structure.
  2. Syntactic Dependency Tree Generator: This component takes an image (encoded by a CNN) and a partial dependency tree (encoded by a Tree-LSTM) as input. It then generates a full syntactic dependency tree for the caption. This generator uses attention mechanisms and GRU networks to decode the tree structure from root to leaf.
  3. Caption Generator: This component takes the full syntactic dependency tree generated in the previous step and "flattens" it into a natural language caption using an LSTM decoder. This ensures that the generated caption reflects the controlled syntactic structure.
  4. Dependency Based Evaluation Metric (DBEM): To evaluate control, a new metric is proposed. DBEM measures the overlap of syntactic dependencies between the input partial dependency tree and the dependency tree of the generated caption, offering a more direct assessment of structural control than traditional n-gram based metrics.

Datasets

The authors use two standard image captioning datasets for their experiments:

  • Microsoft COCO: A large-scale dataset for image captioning.
  • Flickr30k: Another widely used dataset for image captioning.

They also introduce two ways of creating partial dependency trees for evaluation:

  • testgold: Partial dependency trees sampled from reference captions.
  • testpred: Partial dependency trees generated from images themselves, using a trained syntactic dependency classifier.

Key Insights and Ideas

  • Syntactic Structure is Crucial for Control: The results strongly suggest that controlling syntactic structure, in addition to words, significantly improves the controllability of image captions. The DBEM scores demonstrate a substantial increase when full partial dependency trees are provided as input.
  • Explicitly Learning Structure Matters: SDSAM, by explicitly learning to generate syntactic dependency trees as an intermediate representation, outperforms a baseline model (NIC) in controllability, especially on semantic relation-based metrics like SPICE. This highlights the benefit of an intermediate structural representation.
  • Words Alone are Insufficient for Full Structural Control: While providing words as control signals improves DBEM scores, they are not enough to fully specify syntactic structures. This underscores the necessity of explicit structural information.
  • Improved Semantic Understanding: The SDSAM model shows better performance on SPICE, a metric that evaluates semantic relationships, suggesting that better structural control leads to a deeper semantic understanding of the image content in the generated captions.

Key Results

  • Controllability: The DBEM scores show a dramatic increase when partial dependency trees are used as control signals, reaching scores of around 70% on testgold for SDSAM, compared to very low scores when no control is given.
  • Quality: While SDSAM performs comparably to NIC on n-gram based metrics (BLEU, METEOR, ROUGE, CIDEr), it achieves significantly better performance on SPICE, indicating improved semantic accuracy.
  • Diversity: The case study demonstrates that providing different partial dependency trees can lead to considerably different, yet still accurate, captions for the same image, suggesting improved caption diversity.

Limitations and Future Work

  • Dataset Creation: The paper acknowledges that the testpred dataset, where partial dependency trees are generated from images, doesn't significantly impact caption quality compared to testgold. This suggests that the accuracy of the predicted dependency trees might be a bottleneck.
  • Computational Cost: While not explicitly detailed as a limitation, the two-stage pipeline (dependency tree generation then caption generation) might introduce computational overhead.
  • Future Work: The authors suggest that further improvements in predicting accurate syntactic dependency trees from images could enhance controllability. Additionally, exploring more complex control signals or integrating this approach with other control mechanisms could be fruitful avenues.

Summary updated

BibTeX
@inproceedings{zhong2021leveraging,
  author = {Wenjie Zhong and Yusuke Miyao},
  title = {{Leveraging Partial Dependency Trees to Control Image Captions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure