Attend More Times for Image Captioning

Jiajun Du, Yu Qin, Hongtao Lu, Yonghua ZhangarXiv 2018 · OpenReview

Abstract: Most attention-based image captioning models attend to the image once per word. However, attending once per word is rigid and is easy to miss some information. Attending more times can adjust the attention position, find the missing information back and avoid generating the wrong word. In this paper, we show that attending more times per word can gain improvements in the image captioning task, without increasing the number of parameters. We propose a flexible two-LSTM merge model to make it convenient to encode more attentions than words. Our captioning model uses two LSTMs to encode the word sequence and the attention sequence respectively. The information of the two LSTMs and the image feature are combined to predict the next word. Experiments on the MSCOCO caption dataset show that our method outperforms the state-of-the-art. Using bottom up features and self-critical training method, our method gets BLEU-4, METEOR, ROUGE-L, CIDEr and SPICE scores of 0.381, 0.283, 0.580, 1.261 and 0.220 on the Karpathy test split.

The "One-Look" Bottleneck

In standard image captioning, models typically follow a "one word, one look" approach: for every word predicted, the model performs exactly one pass of attention over the image. This is a rigid constraint. Human vision is naturally iterative; when describing a scene, we often glance at several areas or re-examine specific objects before selecting the most descriptive word. If a model's single glance is slightly off-target, it lacks the mechanism to correct itself, often leading to missed details or the hallucination of incorrect objects.

A Two-LSTM Merge Model

To address this, we propose a novel two-LSTM merge model that decouples the process of tracking sentence structure from the process of visual attention. This architecture allows the system to "think" more about the visual evidence before committing to a word.

Conceptual overview of the 'Attend More Times' process.

The model consists of three primary components:

  1. Language LSTM: This component focuses on the linguistic history and grammar of the sentence. It encodes the sequence of words generated so far but does not receive image features directly.
  2. Attention Module & Attention LSTM: This is where the iterative refinement occurs. For a single word prediction, this module can run for multiple iterations ($M$). In each step, it uses the current state of the Language LSTM and its own previous state to refine its focus on the image.
  3. Output Module: A two-layer neural network merges the hidden states of both LSTMs ($h_i^1$ and $h_j^2$) along with global image features to predict the next word.
Detailed architecture of the two-LSTM merge model showing the flow between components.

By separating these roles, the Attention LSTM can advance multiple times while the Language LSTM remains at the same word index. Remarkably, this performance boost is achieved without increasing the total number of parameters, as the model reuses the same weights for each additional attention step.

Experimental Results

We evaluated the model on the MSCOCO dataset using the popular Karpathy split and "Bottom-Up" features (object-level features from Faster R-CNN).

Our "Attend More Times" approach consistently outperformed standard "Attend Once" baselines. On the test split, the model achieved a CIDEr score of 1.261 and a BLEU-4 score of 0.381, surpassing the previous state-of-the-art "Up-Down" model. Qualitative analysis confirmed that the model is better at identifying fine-grained details—for example, correctly identifying a "tall tower" where a standard model might only see a generic "building."

Conclusion and Future Work

Our findings indicate that attending 2 or 3 times per word provides the optimal balance of descriptive accuracy and computational efficiency. While the current model uses a fixed number of iterations, future work could involve a dynamic controller to decide how many times to attend for each specific word, spending more "cognitive effort" on complex nouns and less on functional grammar like "the" or "a."

Summary updated

Abstract

Most attention-based image captioning models attend to the image once per word. However, attending once per word is rigid and is easy to miss some information. Attending more times can adjust the attention position, find the missing information back and avoid generating the wrong word. In this paper, we show that attending more times per word can gain improvements in the image captioning task, without increasing the number of parameters. We propose a flexible two-LSTM merge model to make it convenient to encode more attentions than words. Our captioning model uses two LSTMs to encode the word sequence and the attention sequence respectively. The information of the two LSTMs and the image feature are combined to predict the next word. Experiments on the MSCOCO caption dataset show that our method outperforms the state-of-the-art. Using bottom up features and self-critical training method, our method gets BLEU-4, METEOR, ROUGE-L, CIDEr and SPICE scores of 0.381, 0.283, 0.580, 1.261 and 0.220 on the Karpathy test split.
BibTeX
@misc{du2018attend,
  author = {Jiajun Du and Yu Qin and Hongtao Lu and Yonghua Zhang},
  title = {{Attend More Times for Image Captioning}},
  year = {2018},
  eprint = {1812.03283},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.03283},
}
Figure