Abstract
Traditional image captioning models typically produce single, short sentences that fail to capture the complexity of a visual scene. This paper introduces a novel framework for generating long, coherent, and diverse paragraphs. By integrating a Global Topic Vector to capture "meta-concepts" and Coherence Vectors to ensure smooth transitions between sentences, the model produces more human-like narratives. Furthermore, the generation process is formulated as a Variational Autoencoder (VAE), allowing for the production of multiple diverse descriptions for a single image. The model achieves state-of-the-art results on the Stanford Image-Paragraph and Amazon Product-Review datasets.
Introduction
While standard image captioning has seen significant progress, generating multi-sentence paragraphs remains a challenge. Humans naturally describe scenes with a cohesive narrative flow, moving beyond simple object listing. Simply concatenating short captions often results in disjointed text. This research addresses three key requirements for high-quality paragraph generation: global consistency (the "big picture"), local coherence (logical transitions), and diversity (multiple valid interpretations).
Methodology
The proposed model builds upon a hierarchical Recurrent Neural Network (RNN) consisting of a Topic Generation Net and a Sentence Generation Net. Three primary innovations are introduced to this architecture:
1. Global Topic Vector
To move beyond isolated sentence topics, the model computes a weighted summary of all generated sentence topics, termed the Global Topic Vector. This allows the model to recognize higher-level "meta-concepts"—for example, identifying a "city street" rather than just "cars" and "buildings."
2. Coherence Vectors and the Coupling Unit
To ensure narrative flow, the hidden state from the end of a previous sentence is passed into a "Coupling Unit." This unit fuses the previous context with the current sentence's topic and the global topic, forcing the model to maintain logical continuity.
3. Variational Autoencoder (VAE) Formulation
To account for the inherent ambiguity in image description, the authors cast the generation process as a VAE. By sampling from a learned latent space, the model can generate a variety of diverse, yet accurate, paragraphs for the same input image. This approach was found to be more stable and effective than Generative Adversarial Networks (GANs) for discrete text generation.
Datasets and Evaluation
The model was evaluated on two distinct benchmarks:
- Stanford Image-Paragraph Dataset: 19,551 images with detailed human-annotated paragraphs.
- Amazon Product-Review Dataset: Adapted to generate product reviews based on an image and a target star rating, testing the model's ability to handle sentiment conditioning.
Key Results and Insights
The VAE-based model significantly outperformed existing baselines, including RTT-GAN and standard hierarchical models, across all standard metrics (BLEU, METEOR, and CIDEr).
On the Stanford dataset, the model achieved a CIDEr score of 20.93, a substantial improvement over the RTT-GAN baseline of 16.87. Qualitatively, the model demonstrated a superior ability to synthesize scene-level information. For instance, it correctly identifies a "city street" or a "bus stop" where baseline models merely list objects.
Ablation studies confirmed that Coherence Vectors are essential for natural transitions; without them, the model produced abrupt topic shifts. Additionally, the sentiment conditioning on the Amazon dataset showed that the model could successfully adapt its vocabulary (e.g., "great" vs. "inconvenient") based on the desired star rating.
Conclusion
This work demonstrates that generating high-quality paragraphs from images requires explicit modeling of both global context and local transitions. By utilizing a VAE framework, the model achieves both coherence and diversity, setting a new standard for visual storytelling. Future work will explore extending these techniques to generate even longer narratives and full stories.
Summary updated