Learning Image Captioning as a Structured Transduction Task

Davide Bacciu, Davide SerramazzaEANN 2022 · OpenReview

Abstract

Image captioning is a task typically approached by deep encoder-decoder architectures, where the encoder component works on a flat representation of the image while the decoder considers a sequential representation of natural language sentences. As such, these encoder-decoder architectures implement a simple and very specific form of structured transduction, that is a generalization of a predictive problem where the input data and output predictions might have substantially different structures and topologies. In this paper, we explore a generalization of such an approach by addressing the problem as a general structured transduction problem. In particular, we provide a framework that allows considering input and output information with a tree-structured representation. This allows taking into account the hierarchical nature underlying both images and sentences. To this end, we introduce an approach to generate tree-structured representations from images along with an autoencoder working with this kind of data. We empirically assess our approach on both synthetic and realistic tasks.
BibTeX
@inproceedings{bacciu2022learning,
  author = {Davide Bacciu and Davide Serramazza},
  title = {{Learning Image Captioning as a Structured Transduction Task}},
  booktitle = {EANN},
  year = {2022},
  url = {https://openreview.net/forum?id=F_Hj7s8qmN-},
}

Loading related content...

Figure