Guiding Attention using Partial-Order Relationships for Image Captioning

Murad Popattia, Muhammad Rafi, Rizwan Qureshi, Shah NawazWorkshops @ CVPR 2022 · DOI · arXiv

Abstract

This paper introduces a novel image captioning framework that replaces standard symmetric similarity measures with partial-order relationships to better align visual and textual data. By organizing images, captions, and topics into a hierarchical embedding space—where images "envelop" captions, which in turn "envelop" topics—the authors create a guided attention mechanism that produces more semantically precise and contextually aware descriptions.

The Semantic Gap in Image Captioning

Image captioning requires bridging computer vision and natural language processing. Most contemporary systems utilize attention mechanisms to focus on specific image regions during word generation. However, these models often struggle to capture the underlying semantic hierarchy of a scene.

Traditionally, researchers have used symmetric distance metrics, such as cosine similarity, to align images and text in a shared space. The authors argue this approach is fundamentally flawed: an image is a rich, high-level container of information, while a caption is a specific subset of that information, and a "topic" (e.g., "dog" or "grass") is even more granular. Treating these entities as equals ignores their natural hierarchy. The challenge lies in embedding this hierarchical structure directly into the model's architecture to guide attention more effectively.

Methods: The T-OE-ATT Framework

The proposed pipeline centers on a Retrieval Model that enforces a partial-order relationship across visual and textual features.

The T-OE-ATT model architecture, featuring a retrieval model for partial-order embeddings and a dual-LSTM decoder.
  1. Feature Extraction & Topic Classification: Visual features are extracted using a ResNet-152 backbone, while a Latent Dirichlet Allocation (LDA) model identifies high-level topics within the image.
  2. Ordered Embedding Space: This core innovation replaces simple similarity with a hierarchy: Image $\succeq$ Caption $\succeq$ Topics. This partial order ensures that if an image contains a specific topic, the image's vector "covers" the topic's vector in the embedding space.
  3. Dual-LSTM Decoder: Captioning is handled by two branches:
  4. Guiding-LSTM: Processes hierarchical embeddings to provide a semantic "map."
    • Core-LSTM: Manages the actual word generation.
  5. Guided Attention: The model leverages context from ordered embeddings to drive two attention types:
  6. Spatial Attention: Determining where in the image to focus based on the hierarchy.
  7. Temporal Attention: Deciding when to focus on visual features versus linguistic flow.

Datasets and Evaluation

  • MSCOCO: The primary benchmark for training and evaluation, utilizing the "Karpathy" splits (118k training, 5k testing images).
  • Flickr30k: Used to validate sub-components, specifically the topic classifier and the retrieval model's caption ranking capabilities.

Key Insights

  • Asymmetry as a Feature: By acknowledging that an image contains more information than a caption, the partial-order constraint prevents the model from over-simplifying the relationship between pixels and words.
  • Embeddings as a "North Star": Rather than feeding raw topic labels into the decoder, the model uses the position of these topics within the learned hierarchy to guide attention blocks.
  • Dynamic Weighting: The model learns to balance spatial and temporal attention. As training progresses, the model naturally relies more on spatial (visual) cues to fill in specific details.

Results and Qualitative Analysis

The proposed T-OE-ATT (Topic-Ordered Embedding Attention) model achieved competitive results on the MSCOCO dataset, reaching a CIDEr score of 115.9 and a METEOR score of 28.4, outperforming established models like RF-Net and HAN.

A complex street scene where the model leverages partial-order context to distinguish between objects and their specific actions.

Ablation studies demonstrated that moving from raw topics to ordered embeddings (T-OE) resulted in a significant jump in CIDEr score (from 80.2 to 114.4). Qualitatively, the model showed a superior grasp of action-object associations, such as correctly identifying "parked" versus "sitting" buses by leveraging the partial-order context.

Limitations and Future Work

Despite its success, the system's performance is heavily dependent on the accuracy of the initial topic classifier. If the LDA model misses a key element, the guidance for the attention mechanism becomes flawed. Additionally, the model occasionally confuses semantically similar objects (e.g., "bus" vs. "truck") due to their proximity in the embedding space. Future research may involve incorporating partial-order embeddings into Transformer-based architectures to further enhance semantic grounding.

Summary updated

BibTeX
@inproceedings{popattia2022guiding,
  author = {Murad Popattia and Muhammad Rafi and Rizwan Qureshi and Shah Nawaz},
  title = {{Guiding Attention using Partial-Order Relationships for Image Captioning}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  doi = {10.1109/cvprw56347.2022.00513},
}
Figure