Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization

Nicholas Moratelli, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

TL;DR

Direct CLIP-Based Optimization (DiCO) is a novel training paradigm for image captioning that moves away from unstable reinforcement learning. By distilling rewards from high-quality evaluators like CLIP-Score directly into the model's objective, DiCO produces more descriptive, human-aligned captions while avoiding the "reward hacking" and model collapse common in traditional fine-tuning methods.

What Problem Does This Paper Address?

The standard way to train image captioners involves two steps: first, "teacher forcing" (predicting the next word based on ground-truth text), followed by fine-tuning with Self-Critical Sequence Training (SCST). SCST uses reinforcement learning to maximize specific metrics, traditionally n-gram overlap scores like CIDEr.

However, researchers are increasingly moving toward modern metrics like CLIP-Score and PAC-Score, which use multimodal embedding spaces to measure how well a caption actually matches an image. The problem is that SCST is notoriously unstable when optimizing these modern metrics. As shown in Figure 1, models often "hack" the reward—generating repetitive, ungrammatical gibberish (e.g., "a rusted rusty rusty...") that scores high on the metric but is useless to humans.

Methods

The authors propose DiCO, an "RL-free" alignment strategy inspired by Direct Preference Optimization (DPO) used in Large Language Models.

  1. Reward Distillation: Instead of using a separate reinforcement learning loop, DiCO distills a reward model directly into the captioner. It samples multiple candidate captions for an image and uses an external evaluator (like CLIP) to identify a "winner" and several "losers."
  2. Weighted Classification: The model is trained to increase the probability of the "winner" relative to the "losers" by solving a weighted classification problem. This mimics the pairwise quality relations found by the evaluator.
  3. KL-Regularization: To prevent the model from drifting into nonsensical "hacked" territory, DiCO includes a per-token KL-penalty. This ensures the fine-tuned model stays close to its original pre-trained state, maintaining linguistic fluency.

Datasets

The primary experiments were conducted on the COCO dataset using the standard Karpathy splits. To prove the method's robustness, the authors also evaluated generalization on six additional benchmarks:

  • nocaps (out-of-domain objects)
  • VizWiz (images taken by people who are blind)
  • TextCaps (images with text)
  • Conceptual Captions 3M (CC3M)
  • FineCapEval (fine-grained descriptions)
  • Flickr30k

Key Insights and Ideas

  • Stability over Reinforcement: DiCO demonstrates that you don't need complex policy gradient methods to align a vision-language model. Direct optimization is significantly more stable and less prone to the "catastrophic forgetting" of linguistic rules.
  • The Importance of $\beta$: The regularization parameter $\beta$ is crucial. A higher $\beta$ keeps the model grounded in its pre-trained knowledge, while a lower $\beta$ allows it to optimize more aggressively for the target metric. The authors found $\beta = 0.2$ to be the "sweet spot" for balancing fluency and descriptiveness.
  • Avoiding the Repetition Trap: Unlike SCST, which often collapses into repetitive n-grams to maximize scores, DiCO's preference-based learning encourages more varied and detailed descriptions.

Key Results

DiCO achieves state-of-the-art performance across several dimensions:

  • Modern Metrics: It significantly outperforms SCST and other CLIP-based optimization methods on PAC-S and CLIP-S.
  • Human Preference: In a user study, captions generated by DiCO were preferred over 50% of the time when compared to powerful models like BLIP-2 and SCST-tuned variants.
  • Qualitative Superiority: As seen in Figure 3, DiCO generates captions that are more descriptive and grammatically correct than SCST, which often produces repetitive errors when pushed to optimize for PAC-S.

Limitations and Future Work

Despite its strengths, the authors acknowledge several limitations:

  • Factual Errors: The model can still make factual mistakes, such as misidentifying objects (e.g., calling balloons "kites").
  • Entity Recognition: It may fail to recognize specific landmarks or brands (e.g., the Taj Mahal) if they were not prominent in the training data.
  • Hallucination: When the subject of an image is highly uncertain, the model may rely on its learned priors and "hallucinate" details.

Future work could involve incorporating open-world knowledge or more robust factual grounding to address these descriptive inaccuracies.

Summary updated

Abstract

The conventional training approach for image captioning involves pre-training a network using teacher forcing and subsequent fine-tuning with Self-Critical Sequence Training to maximize hand-crafted captioning metrics. However, when attempting to optimize modern and higher-quality metrics like CLIP-Score and PAC-Score, this training method often encounters instability and fails to acquire the genuine descriptive capabilities needed to produce fluent and informative captions. In this paper, we propose a new training paradigm termed Direct CLIP-Based Optimization (DiCO). Our approach jointly learns and optimizes a reward model that is distilled from a learnable captioning evaluator with high human correlation. This is done by solving a weighted classification problem directly inside the captioner. At the same time, DiCO prevents divergence from the original model, ensuring that fluency is maintained. DiCO not only exhibits improved stability and enhanced quality in the generated captions but also aligns more closely with human preferences compared to existing methods, especially in modern metrics. Additionally, it maintains competitive performance in traditional metrics. Our source code and trained models are publicly available at https://github.com/aimagelab/DiCO.
BibTeX
@inproceedings{moratelli2024revisiting,
  author = {Nicholas Moratelli and Davide Caffagni and Marcella Cornia and Lorenzo Baraldi and Rita Cucchiara},
  title = {{Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization}},
  booktitle = {BMVC},
  year = {2024},
}

Loading related content...

Figure