Image Embedding Sampling Method for Diverse Captioning

Sania Waheed, Na Min AnEMNLP 2025

TL;DR

This paper introduces HBoP, a novel framework that enhances image captioning diversity without requiring additional model training. By intelligently sampling image embeddings from segmented regions, HBoP enables smaller, efficient Visual-Language Models (VLMs) to generate captions that are comparable in diversity and relevance to much larger, state-of-the-art models. This is crucial for making advanced captioning capabilities accessible on resource-constrained devices.

What Problem Does This Paper Address?

Current state-of-the-art Visual-Language Models (VLMs) for image captioning, while impressive, often come with a significant computational cost, making them impractical for applications on mobile devices or other resource-limited platforms. On the other hand, smaller VLMs are more efficient but tend to produce generic captions that overlook fine-grained details, lacking the richness and specificity of human-generated descriptions. The challenge is to achieve high caption diversity and detail without resorting to massive models or extensive retraining.

Methods

The core of HBoP lies in its "Hierarchical Bags of Phrases" (HBoP) approach, which leverages a pre-trained segmentation model (like SAM) and a pre-trained captioning model (like BLIP). The process involves three main modules:

  1. Image Segmentation Module (ISM): This module uses a segmentation model to identify distinct regions within an image. It then extracts patch embeddings from the visual encoder's last layer, where self-attention has already captured contextual information.
  2. Hierarchical Composition Module (HCM): This module intelligently selects and organizes these patch embeddings to control caption granularity. It creates three levels of representations:
  3. Global/Fine-grained: Selects the largest segmentation masks.
  4. Regional: Uses K-means clustering on segmentation masks to group related regions.
  5. Fine-grained: Identifies remaining smaller regions. These selected embeddings are then concatenated and zero-padded to maintain a consistent input shape for the captioning module.

  6. Image Captioning Module (ICM): A pre-trained VLM (specifically, BLIP fine-tuned for captioning) is used to generate captions based on the hierarchically sampled embeddings. The process is repeated for each level of embeddings to produce multiple diverse captions.

Crucially, HBoP is a "training-free" framework, meaning it doesn't require additional fine-tuning of the underlying VLM or segmentation models.

Datasets

The paper evaluates HBoP on the following standard image captioning datasets:

  • MSCOCO: (Lin et al., 2014)
  • Flickr30k: (Young et al., 2014)
  • Nocaps: (Agrawal et al., 2019)

Key Insights and Ideas

  • Leveraging Segmentation for Diversity: The paper demonstrates that by intelligently sampling embeddings from segmented image regions, rather than just using the whole image, significant gains in caption diversity can be achieved. This is a key insight into how to guide caption generation towards more varied outputs.
  • Hierarchical Embeddings for Granularity Control: The hierarchical composition of embeddings allows for control over the level of detail in the generated captions, from global scene descriptions to more specific regional details. This provides a structured way to explore different aspects of an image.
  • Training-Free Enhancement: The fact that HBoP achieves these improvements without any additional training of the VLM is a major advantage. It means existing, powerful VLMs can be readily enhanced with this method, making it highly practical.
  • Balancing Diversity and Relevance: A critical finding is that HBoP successfully balances increased diversity with maintained semantic integrity and relevance to the image. This is often a trade-off in captioning, and HBoP appears to navigate it effectively.

Key Results

  • Diversity Scores: HBoP achieved a Div-2 score of 0.735 on MSCOCO, 0.750 on Flickr30k, and 0.748 on Nocaps, outperforming many baselines and approaching human-annotated "gold" captions. It also showed competitive performance in terms of pairwise cosine distance (PCD).
  • Relevance and Semantic Integrity: HBoP maintained strong image-caption relevancy and semantic integrity, comparable to or better than existing methods, as measured by SBERT and CLIP-Score.
  • Efficiency: HBoP uses significantly fewer parameters than large LLM-based VLMs while achieving comparable or better diversity, offering a strong trade-off between model size and performance. For instance, it uses 4x to 13x fewer parameters than models like BLIP-2, Honeybee, and LLaVA.
  • Comparison to Cropping: A comparison with a baseline that directly crops segmented regions showed that while cropping can improve diversity, it often sacrifices relevance, whereas HBoP's embedding sampling preserves contextual understanding.

Limitations and Future Work

  • Bounding Box Approximations: The current implementation uses bounding box approximations of segmentation masks. The authors suggest exploring the use of full, irregular-shaped segmentation masks for potentially capturing finer details.
  • Factual Diversity vs. Cultural Diversity: HBoP primarily enhances factual diversity by focusing on distinct image regions. It may be less effective for domains requiring cultural diversity, which often relies on external cultural knowledge beyond visual features.
  • Future Work: The authors suggest exploring the use of full segmentation masks and investigating how HBoP can be extended to capture cultural diversity. They also note that the framework is flexible and can be integrated with different segmentation models, including faster alternatives like FastSAM, to further reduce computational overhead.

Summary updated

BibTeX
@inproceedings{waheed2025image,
  author = {Sania Waheed and Na Min An},
  title = {{Image Embedding Sampling Method for Diverse Captioning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure