Object Relation Attention for Image Paragraph Captioning

Li-Chuan Yang, Chih-Yuan Yang, Jane Yung-jen HsuAAAI 2021 · DOI

Abstract

Image paragraph captioning aims to generate detailed, multi-sentence descriptions of visual scenes. A significant challenge in this task is maintaining consistency and capturing the complex relationships between objects. This paper proposes a novel Object Relation Attention mechanism that explicitly encodes spatial relationships into visual features. By focusing on overlapping object pairs and employing asymmetric encoding, the model produces more coherent and descriptive paragraphs, achieving state-of-the-art performance on the Stanford Paragraph dataset.

Introduction

While standard image captioning typically produces a single, concise sentence, image paragraph captioning seeks to tell a more comprehensive story. However, generating multiple sentences introduces the risk of "hallucinations" or logical contradictions between sentences. Existing models often process detected objects in isolation, leaving the language model to infer relationships during the generation phase. This paper addresses this by integrating relational information directly into the visual features before the language generation phase begins.

Methodology

The proposed framework augments standard object features (extracted via Faster R-CNN) with relational data through several key components:

  1. Spatial Coherence Filter: To manage computational complexity and reduce noise, the model selectively considers pairs of objects with overlapping bounding boxes. In the Stanford dataset, these account for roughly 40% of all possible pairs.
  2. Relation Encoding Network: This module concatenates features of overlapping objects asymmetrically. This asymmetry is crucial for capturing directional relationships, such as "a person wearing a hat" versus "a hat wearing a person."
  3. Relation Attention Network: This network learns to weight the importance of object pairs based on their prevalence and relevance in the training corpus.
  4. Feature Fusing and Generation: Relation-aware features are fused with original visual features and fed into a hierarchical LSTM architecture for paragraph generation.
The overall architecture of the proposed Object Relation Attention framework, showing the flow from input image to generated paragraph.

The model is optimized using Self-Critical Sequence Training (SCST), directly targeting evaluation metrics like CIDEr rather than simple word-prediction accuracy.

Experiments and Results

The model was primarily evaluated on the Stanford Paragraph Dataset, which contains approximately 19,500 images with human-annotated paragraphs. Secondary evaluations were performed on MS COCO to test generalization to single-sentence captioning.

The proposed method established a new state-of-the-art on the Stanford benchmark, achieving a CIDEr score of 33.38, a substantial improvement over previous hierarchical (13.52) and coherence-based (19.95) models. Qualitative analysis confirms that the model maintains better consistency across sentences, accurately preserving object relationships throughout the generated story.

Examples of object pairs identified by the model along with their corresponding attention scores, illustrating how the system prioritizes meaningful spatial relations.

Discussion and Conclusion

Several key insights emerged from this work. First, spatial overlap serves as an effective proxy for semantic meaning, allowing the model to identify relevant relations without explicit relationship labels. Second, asymmetric encoding is vital for aligning visual features with the subject-verb-object structure of human language. Finally, encoding these relations at the feature level provides a clearer signal to the language model, significantly reducing contradictions.

Future work may explore handling relationships between distant objects and incorporating explicit semantic predicates to further refine the descriptive quality of the generated paragraphs.

Summary updated

BibTeX
@inproceedings{yang2021object,
  author = {Li-Chuan Yang and Chih-Yuan Yang and Jane Yung-jen Hsu},
  title = {{Object Relation Attention for Image Paragraph Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16423},
}
Figure