Abstract
Generating coherent, multi-sentence paragraphs from images remains a significant challenge in computer vision and natural language processing. Standard image captioning models often produce generic, repetitive text due to the difficulty of bridging raw pixel data with high-level symbolic reasoning. This paper introduces the Sparse Graph-to-Sequence Transformer (SGST), a model that leverages scene graphs and a specialized sparse attention mechanism ($\alpha$-entmax) to generate detailed and descriptive paragraphs. By filtering out irrelevant visual noise, SGST achieves a 13.3% improvement in CIDEr scores over previous state-of-the-art methods.
Introduction
While AI has mastered short image captions, generating long, coherent narratives requires an understanding of the complex relationships between objects in a scene. When humans describe an image, they don't merely list objects; they synthesize structural relationships—for instance, noting that a "muffin" sits atop a "napkin." Standard models struggle with this "vision-to-sequence" gap, often resulting in repetitive or generic outputs. This research addresses the problem by treating image-to-paragraph generation as a graph-to-text task, aiming for "brief and coherent stories" rather than simple labels.
Methodology: The Sparse Graph-to-Sequence Transformer (SGST)
The proposed SGST architecture consists of two primary components designed to handle structural data efficiently.
1. Scene Graph Encoder
Rather than processing raw image features directly, the system extracts a "scene graph" where nodes represent objects or attributes and edges represent their relationships. The Transformer architecture is adapted to operate on these graph structures, capturing the underlying semantics of the scene.
2. Sparse Attention ($\alpha$-entmax)
The core innovation of SGST is the replacement of the standard softmax attention with $\alpha$-entmax. Standard Transformers assign a non-zero weight to every node in a graph, which introduces noise in complex scenes. In contrast, $\alpha$-entmax can assign exactly zero probability to irrelevant nodes, allowing the model to focus exclusively on the most salient elements. This sparsity acts as an inductive bias, mimicking human selective attention.
Experimental Setup
The model was evaluated using the Image Paragraph Dataset, which contains 19,551 image-paragraph pairs. Unlike standard datasets where captions are short, these paragraphs average 67.5 words, presenting a significant descriptive challenge. The researchers utilized both Ground Truth (GT) scene graphs from the Visual Genome dataset and automatically generated graphs (via Faster R-CNN) to assess the model's robustness.
Results and Discussion
The SGST model demonstrated superior performance across multiple metrics, most notably achieving a CIDEr score of 28.50, representing a 13.3% improvement over the previous state-of-the-art (25.15).
Qualitative analysis reveals that the sparse model produces more "coherent and brief" paragraphs. By utilizing $\alpha$-entmax, the model avoids the wordiness and imprecision often found in softmax-based outputs. Furthermore, linguistic analysis showed an increase in the use of verbs and pronouns, suggesting that the model is better at capturing actions and maintaining referential continuity (e.g., correctly using "it" to refer to a previously mentioned "banana").
Conclusion and Future Work
The SGST framework highlights the power of combining symbolic reasoning (via scene graphs) with sparse attention mechanisms for long-form text generation. While the model shows significant gains, its performance remains dependent on the quality of the initial scene graph. Future research will explore deeper graph reasoning techniques, such as multi-hop reasoning, and compare the Transformer-based approach with Graph Convolutional Networks (GCNs) to further refine structural sparsity.
Summary updated