Text Embedding Bank for Detailed Image Paragraph Captioning

Arjun Gupta, Zengming Shen, Thomas S. HuangAAAI 2021 · DOI

Introduction

Generating detailed paragraphs to describe images is a significantly more complex task than standard image captioning. While current models excel at identifying single objects or simple actions (e.g., "a dog on a rug"), they often struggle to produce coherent, multi-sentence descriptions. Researchers have identified two primary bottlenecks in this process: insufficient supervision for the image encoder and the difficulty of maintaining long-term dependencies in recurrent neural networks (RNNs).

To address these challenges, researchers have introduced the Text Embedding Bank (TEB), a plug-and-play module that significantly improves how AI generates detailed paragraphs. By providing the model with a "global" understanding of the entire paragraph during training, TEB helps the system extract richer visual features and maintain better coherence over long sequences.

The Text Embedding Bank (TEB) Framework

The core innovation of this approach is the use of a paragraph vector model to map an entire ground-truth paragraph into a single, fixed-length vector. This vector serves as a comprehensive representation of the paragraph's content and structure.

Architecture of the proposed TEB framework for image paragraph captioning.

The system operates in two distinct phases:

  1. Training (Global Deep Supervision): The image encoder is trained not only to identify local objects but also to predict the global paragraph vector (TEB'). By minimizing the L1 loss between the predicted TEB' and the actual ground-truth TEB, the model receives a strong supervisory signal that forces the encoder to capture complex visual details necessary for long-form descriptions.
  2. Inference (Distributed Memory): During generation, the model produces a predicted TEB' from the input image. This vector is concatenated with standard visual features and fed into the RNN decoder. In this context, TEB' acts as a "distributed memory," providing a constant thematic anchor that helps the RNN maintain global coherence and avoid "forgetting" the initial context as the paragraph grows.

Key Innovations

The TEB module introduces two critical concepts to the captioning pipeline:

  • Global vs. Local Feedback: Traditional word-level loss is often too "noisy" to effectively train an image encoder for paragraphs. By supervising the encoder with a representation of the entire paragraph, the model learns to prioritize visual features that contribute to the overall narrative.
  • TEB as an Anchor: In the decoder phase, TEB' serves as a stabilizer. As a fixed-length vector representing the intended output, it helps alleviate the vanishing gradient problem common in RNNs, acting as a "cheat sheet" for long-term context.

Experimental Results

The effectiveness of the TEB module was evaluated on the Stanford Visual Genome dataset, the standard benchmark for image paragraph captioning. The module was integrated into several existing architectures, including Transformer-based and Diversity-based models.

Performance comparison of the TEB module against state-of-the-art methods on the Stanford Visual Genome dataset.

As shown in the results table, the TEB module provided a consistent performance boost across all major metrics. The top-performing configuration, Ours (D + TEB), which combined TEB with a diversity-focused model, achieved a new state-of-the-art (SOTA) on the dataset. Specifically, it reached a METEOR score of 18.93 and a CIDEr score of 32.53, outperforming previous specialized architectures like CAE-LSTM and RTT-GAN.

Conclusion

The Text Embedding Bank represents a significant step forward in detailed image description. By providing global supervision and a distributed memory mechanism, it allows models to bridge the gap between local visual recognition and global narrative coherence. Future work may explore integrating more advanced embeddings from large language models or extending the TEB concept to video paragraph captioning.

Summary updated

BibTeX
@inproceedings{gupta2021text,
  author = {Arjun Gupta and Zengming Shen and Thomas S. Huang},
  title = {{Text Embedding Bank for Detailed Image Paragraph Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17892},
}
Figure