Diverse and Coherent Paragraph Generation from Images

Moitreya Chatterjee, Alexander G. Schwing

Abstract

Traditional image captioning models typically produce single, short sentences that fail to capture the complexity of a visual scene. This paper introduces a novel framework for generating long, coherent, and diverse paragraphs. By integrating a Global Topic Vector to capture "meta-concepts" and Coherence Vectors to ensure smooth transitions between sentences, the model produces more human-like narratives. Furthermore, the generation process is formulated as a Variational Autoencoder (VAE), allowing for the production of multiple diverse descriptions for a single image. The model achieves state-of-the-art results on the Stanford Image-Paragraph and Amazon Product-Review datasets.

Introduction

While standard image captioning has seen significant progress, generating multi-sentence paragraphs remains a challenge. Humans naturally describe scenes with a cohesive narrative flow, moving beyond simple object listing. Simply concatenating short captions often results in disjointed text. This research addresses three key requirements for high-quality paragraph generation: global consistency (the "big picture"), local coherence (logical transitions), and diversity (multiple valid interpretations).

Methodology

The proposed model builds upon a hierarchical Recurrent Neural Network (RNN) consisting of a Topic Generation Net and a Sentence Generation Net. Three primary innovations are introduced to this architecture:

1. Global Topic Vector

To move beyond isolated sentence topics, the model computes a weighted summary of all generated sentence topics, termed the Global Topic Vector. This allows the model to recognize higher-level "meta-concepts"—for example, identifying a "city street" rather than just "cars" and "buildings."

The hierarchical architecture showing the SentenceRNN and the derivation of the Global Topic Vector.

2. Coherence Vectors and the Coupling Unit

To ensure narrative flow, the hidden state from the end of a previous sentence is passed into a "Coupling Unit." This unit fuses the previous context with the current sentence's topic and the global topic, forcing the model to maintain logical continuity.

3. Variational Autoencoder (VAE) Formulation

To account for the inherent ambiguity in image description, the authors cast the generation process as a VAE. By sampling from a learned latent space, the model can generate a variety of diverse, yet accurate, paragraphs for the same input image. This approach was found to be more stable and effective than Generative Adversarial Networks (GANs) for discrete text generation.

Datasets and Evaluation

The model was evaluated on two distinct benchmarks:

  • Stanford Image-Paragraph Dataset: 19,551 images with detailed human-annotated paragraphs.
  • Amazon Product-Review Dataset: Adapted to generate product reviews based on an image and a target star rating, testing the model's ability to handle sentiment conditioning.

Key Results and Insights

The VAE-based model significantly outperformed existing baselines, including RTT-GAN and standard hierarchical models, across all standard metrics (BLEU, METEOR, and CIDEr).

Qualitative comparison between the proposed approach and a hierarchical baseline, highlighting the use of meta-concepts like 'city street'.

On the Stanford dataset, the model achieved a CIDEr score of 20.93, a substantial improvement over the RTT-GAN baseline of 16.87. Qualitatively, the model demonstrated a superior ability to synthesize scene-level information. For instance, it correctly identifies a "city street" or a "bus stop" where baseline models merely list objects.

Ablation studies confirmed that Coherence Vectors are essential for natural transitions; without them, the model produced abrupt topic shifts. Additionally, the sentiment conditioning on the Amazon dataset showed that the model could successfully adapt its vocabulary (e.g., "great" vs. "inconvenient") based on the desired star rating.

Conclusion

This work demonstrates that generating high-quality paragraphs from images requires explicit modeling of both global context and local transitions. By utilizing a VAE framework, the model achieves both coherence and diversity, setting a new standard for visual storytelling. Future work will explore extending these techniques to generate even longer narratives and full stories.

Summary updated

BibTeX
@inproceedings{chatterjee2018diverse,
  author = {Moitreya Chatterjee and Alexander G. Schwing},
  title = {{Diverse and Coherent Paragraph Generation from Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=rJ4xIFZOZS},
}
Figure