Rewarding Only the Words the Image Supports: Grounded Length Correction for Image-Seeded Decoders

Many image captioning and multimodal translation systems tell the text generator about the image only once, before the first word. They do this by setting the generator's starting internal memory (its "initial hidden state") from the image's feature vector. As the sentence grows, that image information fades. The generator then falls back on generic language habits, and a standard search procedure called beam search, which keeps the highest-scoring partial sentences, tends to stop early. It stops early because every extra word multiplies in another probability below one, so longer sentences always score lower.

The usual fix gives every word the same small bonus. That fix cannot tell a word the image actually supports from a fluent word the image does not support. We propose a different correction: give each token a length bonus only in proportion to how much the image raised its probability. We measure this by running the same decoder a second time with a neutral, image-free starting state. The result corrects for short outputs where the image justifies more description, and it withholds the bonus where extra length would only add hallucinated content.

The proposal ties together three findings that have not been connected. The first is that image-seeded decoders forget the image as they go ("drift"), shown in Guiding the Long-Short Term Memory Model for Image Caption Generation. The second is that beam search prefers short outputs and can collapse as the beam widens, shown in Correcting Length Bias in Neural Machine Translation. The third is that image and text vectors occupy separate regions of representation space (the "modality gap"), shown in Mind the Gap. That third finding determines how a fair image-free reference state must be built.

Core Claim

In decoders seeded once with the image, brevity bias is partly a grounding problem and not only an arithmetic one. Early stopping should be most likely at exactly the steps where the image's influence on the next-word distribution has faded.

If that is true, a length correction tied to the image's influence on each word should match a tuned constant word reward on overall caption quality. It should beat that reward where appropriate length varies with the image, and it should add fewer unsupported words. The scoring rule for a candidate sentence $Y=(y_1,\dots,y_T)$ given image $I$ is

$$S(Y)=\sum_{t=1}^{T}\Big[\log p(y_t\mid y_{<t},h_0(I)) + \lambda\,\max\big(0,\ \log p(y_t\mid y_{<t},h_0(I))-\log p(y_t\mid y_{<t},\bar h_0)\big)\Big].$$

The symbols are as follows. The term $h_0(I)$ is the initial hidden state computed from the image. The term $\bar h_0$ is an image-free reference initial state, with the same decoder weights and only the starting state changed. The weight $\lambda$ is a single tuned scalar. The sum ranges over every emitted token, including the end-of-sentence token.

The bracketed bonus is the clipped pointwise gain the image gives each token. Several properties follow from it:

  • A word the image makes no more likely earns no bonus, however fluent it is.
  • Stopping is itself rewarded when the image makes ending the sentence likely.
  • A constant reward $\gamma$ per token, as in Murray and Chiang, is the special case where every token's gain is replaced by the same number.

Research Program

Construct the reference state. This is where the modality gap enters. A zero vector is the obvious image-free state. However, Mind the Gap shows that image embeddings cluster in a narrow cone offset from the origin and from text embeddings. A zero-state counterfactual would therefore mix two effects: "this particular image" and "being in the image region at all." The second effect is a constant shift and carries no content.

We will compare three reference states:

  • The zero vector.
  • The projection of the mean training-image embedding, which keeps the offset but removes image-specific content.
  • The mean-image projection shifted along the gap direction, following the shifting procedure in Mind the Gap.

Our prediction is that the mean-image reference gives a cleaner grounding signal than zero, measured by how well per-token gains track words that appear in the reference captions. This matters because Incorporating Global Visual Features into Attention-based Neural Machine Translation seeds recurrent states through a plain affine map of image features and never asks whether that map's offset distorts the decoder's starting point.

Diagnose the mechanism before claiming the fix. We will train decoders under four ways of supplying the image:

For each decoder we will track two quantities along every beam path:

  • The image gain at each step.
  • A linear probe's ability to recover the image embedding from the hidden state $h_t$.

We then test whether the probability of ending the sentence rises as the gain falls, after controlling for position and for reference length. The prediction is that this relationship is strong for $t=0$ seeding, weaker for gLSTM, and weakest with attention. If it is absent, the grounding view of brevity bias is wrong, and the method reduces to a heuristic.

Build the decoder and keep it cheap. The reference pass shares all weights and differs only in its initial state, so decoding cost roughly doubles and no new parameters are trained. We will tune $\lambda$ on development data with the same perceptron-style length-matching update that Murray and Chiang use for $\gamma$. This keeps the tuning budget identical across methods.

Baselines

Unnormalized beam search at increasing widths. This is the do-nothing reference. On Russian–English, Correcting Length Bias in Neural Machine Translation reports BLEU falling from 24.9 at beam 10 to 3.7 at beam 1000. We will plot the same curve for each captioning decoder from beam 1 to beam 1000. That shows whether image-seeded captioners suffer the collapse at all.

Length division and Jia et al.'s normalizations. These are the simple fixed corrections:

No captioning numbers for the latter were found on Lacuna. We will rerun them on our decoders.

Tuned constant word reward, the strongest competitor. This is the reward of Murray and Chiang, which is reported to hold 25.7 BLEU on Russian–English even at beam 1000. It is the strongest competitor because it eliminates beam collapse with one tuned scalar and no model changes. Our method contains it as a special case.

For a fair comparison, both methods get:

  • One scalar tuned on the same development split.
  • The same update rule and number of tuning passes.
  • The same beam widths and the same trained decoder.

Beating it means two things. First, equal or better CIDEr-D and BLEU-4 on standard splits. Second, a clear gain on the subsets where reference length varies most, together with a lower rate of generated object words absent from the image's annotated objects. For orientation, the original image-seeded captioner reports 27.7 BLEU-4 on MS COCO (Show and tell), and the human reference on the 40-reference split is BLEU-4 0.471 and CIDEr-D 0.910 (Microsoft COCO Captions).

Image-conditioned length predictor. A small regressor predicts each image's reference length from its features. The predicted length then centers a per-image Gaussian normalization. This is the direct per-image version of Jia et al. and the most natural rival design.

Ablations of the method's key ingredient. Two ablations isolate what we claim is new:

  • Replace the per-token image gain with its average over the sentence. This keeps the image-dependence but discards the token-level selectivity.
  • Swap the mean-image reference state for the zero state. This isolates the modality-gap correction.

Why not just…?

Why not just use a tuned constant word reward? This is a real threat. On a corpus like MS COCO, whose collection guidelines push captions toward a narrow length range (Microsoft COCO Captions), one scalar may be nearly optimal, and our method may only tie. Its weakness should show where appropriate length depends on the input. In Zhou et al.'s IKEA data, descriptions run 60–70 words, against roughly ten for standard captions (A Visual Attention Grounding Neural Model for Multimodal Machine Translation). Chen et al. found that pushing length correction past its sweet spot produced repetition (A Semantics-Assisted Video Captioning Model). The length-stratified and mixed-corpus experiments below settle the question. If the constant reward matches us there too, we will report that the grounding term adds nothing.

Why not just predict the length from the image? A length predictor addresses Gap 3 of the research file directly and may recover most of the length benefit. Its limitation is that it controls how many words appear, not which ones. A beam pushed toward a predicted length of 15 can fill the extra slots with plausible but unsupported content. We will therefore compare the two at matched output length, by tuning both to the same mean length, and measure the rate of words with no visual support. If the predictor matches us on that rate, the per-token signal is unnecessary.

Why not just feed the image at every step so it never fades? Continuous guidance attacks drift at its source. Jia et al. show it helps. However, Show and tell reports that per-step image feeding overfit to visual noise, and gLSTM still needed length normalization at decoding. Our diagnostic directly measures how much less drift continuous guidance leaves. Applying our decoder to the gLSTM model shows whether the two are complementary or redundant. A small gain there would be an honest, informative result.

Why not just close the modality gap with a better projection? Mind the Gap finds that fully closing the gap is not optimal. Extending Multi-modal Contrastive Representations finds that single projection layers struggle to align the two spaces. We use the gap only to define a neutral reference state, not as something to eliminate. We will also test the gap-shift knob alone, with no grounding reward, to see whether moving $h_0$ along the gap direction changes end-of-sentence confidence. If shifting alone fixes length bias, that is a simpler method, and we would adopt it.

Proposed Experiments

Data. We train on MS COCO and Flickr30K for captioning, Multi30K for English–German multimodal translation (Multi30K), and MSVD for video. On MSVD, Chen et al. report 62.4 BLEU-4 with a length-modulated training loss (A Semantics-Assisted Video Captioning Model).

Multi30K serves as a control. The source sentence carries most of the information there, so image gains should be small and the method should reduce to a weak word reward. Larger gains on Multi30K than on COCO would suggest we are measuring something other than grounding.

The decisive length test uses a mixed-length evaluation. It combines COCO captions with IKEA product descriptions, on which Zhou et al. report 63.5 BLEU for their translation model. It also splits COCO test images by number of annotated objects. A single scalar must then serve both short and long targets.

Metrics. We report BLEU-4, CIDEr-D, and METEOR. We also report the ratio of output length to reference length per stratum, and the fraction of generated object nouns not among the image's annotated objects.

Decisive checks. We report beam-width curves out to 1000 for every decoder and correction.

The grounding view of brevity bias is supported only if three results hold together:

  • The diagnostic shows end-of-sentence probability rising as the image gain falls for $t=0$ seeding.
  • Our decoder matches the tuned word reward on COCO.
  • On the mixed-length and object-count strata, our decoder beats the word reward and the length predictor in quality while producing fewer unsupported object words.

Two further results would confirm the role of each ingredient. The averaged-gain ablation should recover the length benefit but lose the hallucination benefit. The zero-reference ablation should give noisier gains than the mean-image reference.