ParaCNN: Visual Paragraph Generation via Adversarial Twin Contextual CNNs

Shiyang Yan, Yang Hua, Neil RobertsonarXiv 2020 · OpenReview

Abstract

Image description generation plays an important role in many real-world applications, such as image retrieval, automatic navigation, and disabled people support. A well-developed task of image description generation is image captioning, which usually generates a short captioning sentence and thus neglects many of fine-grained properties, e.g., the information of subtle objects and their relationships. In this paper, we study the visual paragraph generation, which can describe the image with a long paragraph containing rich details. Previous research often generates the paragraph via a hierarchical Recurrent Neural Network (RNN)-like model, which has complex memorising, forgetting and coupling mechanism. Instead, we propose a novel pure CNN model, ParaCNN, to generate visual paragraph using hierarchical CNN architecture with contextual information between sentences within one paragraph. The ParaCNN can generate an arbitrary length of a paragraph, which is more applicable in many real-world applications. Furthermore, to enable the ParaCNN to model paragraph comprehensively, we also propose an adversarial twin net training scheme. During training, we force the forwarding network's hidden features to be close to that of the backwards network by using adversarial training. During testing, we only use the forwarding network, which already includes the knowledge of the backwards network, to generate a paragraph. We conduct extensive experiments on the Stanford Visual Paragraph dataset and achieve state-of-the-art performance.

Introduction

While image captioning has seen massive progress, most models produce a single sentence of fewer than twenty words. This is often insufficient for describing complex scenes with multiple objects and intricate relationships. Visual paragraph generation aims to provide a "fine-grained" description, but it faces two major hurdles: coherence and long-term dependencies. Sentences within a paragraph must be consistent and flow logically. Traditional sequence models like Recurrent Neural Networks (RNNs) struggle to maintain context over long sequences and are difficult to parallelize due to their step-by-step nature.

The authors argue that the complex "memorizing and forgetting" mechanisms of RNNs are a bottleneck and propose that a hierarchical Convolutional Neural Network (CNN) can handle these long-term dependencies more effectively and efficiently.

Methods

The ParaCNN Architecture

Unlike previous hierarchical models that use LSTMs, ParaCNN is built entirely on convolutions. It uses a two-level hierarchy:

  • Paragraph Convolutions (Topic Level): These layers decompose visual features into a sequence of "visual topics." Each topic serves as the seed for a single sentence. Crucially, it uses a "coupling mechanism" where the context of the previously generated sentence informs the generation of the next topic.
  • Sentence Convolutions (Word Level): These layers take a visual topic and decode it into a sequence of words. By using masked 1D convolutions, the model ensures it only looks at past information during the generation process.
The hierarchical architecture of ParaCNN showing Sentence and Paragraph Convolution modules.

Adversarial Twin Net Training

Standard language models only look at the "past" to predict the "next" word. To help the model reason about the "future" of the paragraph, the authors used a "Twin Net" scheme. They train a forward network (normal generation) and a backward network (generating the paragraph in reverse). Using a Wasserstein-GAN (W-GAN) framework, they force the hidden features of the forward network to match the distribution of the backward network’s features. This "adversarial" alignment acts as a form of knowledge transfer, teaching the forward model to produce representations that are useful for the entire paragraph's structure, not just the immediate next word.

The Adversarial Twin Net training scheme utilizing a Generator and Discriminator to align forward and backward hidden features.

Experimental Setup

The model was evaluated on the Stanford Visual Paragraph dataset, derived from the Visual Genome dataset. It consists of 14,575 images for training, 2,487 for validation, and 2,489 for testing. Each image is annotated with a multi-sentence paragraph containing significantly more pronouns, verbs, and diverse vocabulary than standard captioning datasets like MS-COCO.

Results and Discussion

ParaCNN achieved state-of-the-art results across all major language metrics. On the CIDEr metric, ParaCNN reached a score of 21.6, significantly outperforming previous best hierarchical RNN models like RTT-GAN (16.9) and Diverse VAE (20.9).

Beyond accuracy, ParaCNN is significantly more efficient. It uses only 38.27M parameters compared to the 165.65M parameters required by traditional hierarchical RNNs. Training time was reduced from 2506 seconds per epoch (RNN) to 2078 seconds per epoch (ParaCNN).

Key Insights

  • CNNs for Language: CNNs offer fixed-length paths for gradients, making them easier to optimize for long paragraphs compared to RNNs.
  • Parallelism: Because convolutions don't require the hidden state of the previous step to compute the current one during training, the entire paragraph can be processed at once, leading to faster training.
  • Future Context: The adversarial twin net effectively solves the "one-step-ahead" myopia of standard training by aligning with a backward-looking model.

Conclusion

ParaCNN demonstrates that a pure CNN-based hierarchical architecture can outperform complex RNN models in visual paragraph generation. By incorporating future context through an adversarial twin net training scheme, the model achieves superior coherence and detail while maintaining high computational efficiency. Future work may explore integrating self-critical training or external knowledge bases to further enhance text richness.

Summary updated

Abstract

Image description generation plays an important role in many real-world applications, such as image retrieval, automatic navigation, and disabled people support. A well-developed task of image description generation is image captioning, which usually generates a short captioning sentence and thus neglects many of fine-grained properties, e.g., the information of subtle objects and their relationships. In this paper, we study the visual paragraph generation, which can describe the image with a long paragraph containing rich details. Previous research often generates the paragraph via a hierarchical Recurrent Neural Network (RNN)-like model, which has complex memorising, forgetting and coupling mechanism. Instead, we propose a novel pure CNN model, ParaCNN, to generate visual paragraph using hierarchical CNN architecture with contextual information between sentences within one paragraph. The ParaCNN can generate an arbitrary length of a paragraph, which is more applicable in many real-world applications. Furthermore, to enable the ParaCNN to model paragraph comprehensively, we also propose an adversarial twin net training scheme. During training, we force the forwarding network's hidden features to be close to that of the backwards network by using adversarial training. During testing, we only use the forwarding network, which already includes the knowledge of the backwards network, to generate a paragraph. We conduct extensive experiments on the Stanford Visual Paragraph dataset and achieve state-of-the-art performance.
BibTeX
@misc{yan2020paracnn,
  author = {Shiyang Yan and Yang Hua and Neil Robertson},
  title = {{ParaCNN: Visual Paragraph Generation via Adversarial Twin Contextual CNNs}},
  year = {2020},
  eprint = {2004.10258},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.10258},
}
Figure