Structural Semantic Adversarial Active Learning for Image Captioning

Beichen Zhang, Liang Li, Shuhui Wang, Jincan Deng, Zheng-Jun Zha, Qingming Huang

Abstract

Most image captioning models achieve superior performances with the help of large-scale surprised training data, but it is prohibitively costly to label the image captions. To solve this problem, we propose a structural semantic adversarial active learning (SSAAL) model that leverages both visual and textual information for deriving the most representative samples while maximizing the image captioning performance. SSAAL consists of a semantic constructor, a snapshot& caption (SC) supervisor, and a labeled/unlabeled state discriminator. The constructor is designed to generate a structural semantic representation describing the objects, attributes and object relationships in the image. The SC supervisor is proposed to supervise this representation at the word-level and sentence-level in a multi-task learning manner, which directly relates the representation to ground-truth captions and updates it in the caption generating process. Finally, we introduce a state discriminator to predict the sample state and select images with sufficient semantic and fine-grained diversity. Extensive experiments on standard captioning dataset show that our model outperforms other active learning methods and achieves a competitive performance even though selecting a small amount of samples.
BibTeX
@inproceedings{zhang2020structural,
  author = {Beichen Zhang and Liang Li and Li Su and Shuhui Wang and Jincan Deng and Zheng-Jun Zha and Qingming Huang},
  title = {{Structural Semantic Adversarial Active Learning for Image Captioning}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413885},
}

Loading related content...

Figure