Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization

Nicholas Moratelli, Davide Caffagni, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

Abstract

The conventional training approach for image captioning involves pre-training a network using teacher forcing and subsequent fine-tuning with Self-Critical Sequence Training (SCST) to maximize hand-crafted captioning metrics. However, when attempting to optimize modern and higher-quality metrics like CLIP-Score and PAC-Score, this training method often encounters instability and fails to acquire the genuine descriptive capabilities needed to produce fluent and informative captions. In this paper, we propose a new training paradigm termed Direct CLIP-Based Optimization (DiCO). Our approach jointly learns and optimizes a reward model that is distilled from a learnable captioning evaluator with high human correlation. This is done by solving a weighted classification problem directly inside the captioner. At the same time, DiCO prevents divergence from the original model, ensuring that fluency is maintained. DiCO not only exhibits improved stability and enhanced quality in the generated captions but also aligns more closely with human preferences compared to existing methods, especially in modern metrics. Additionally, it maintains competitive performance in traditional metrics.

Introduction

The standard training pipeline for image captioners involves two primary steps: initial "teacher forcing" (predicting the next word based on ground-truth text), followed by fine-tuning with Self-Critical Sequence Training (SCST). SCST typically uses reinforcement learning to maximize n-gram overlap scores like CIDEr.

However, there is a growing shift toward modern metrics like CLIP-Score and PAC-Score, which utilize multimodal embedding spaces to evaluate how well a caption matches an image semantically. SCST is notoriously unstable when optimizing these modern metrics. As shown in the training dynamics, models often "hack" the reward—generating repetitive, ungrammatical gibberish that scores high on the metric but is nonsensical to humans. This instability often leads to model collapse where traditional metrics like CIDEr drop to near zero while trying to optimize for semantic alignment.

Comparison between traditional SCST and the proposed DiCO approach. The training curves illustrate how SCST can lead to metric collapse (e.g., CIDEr dropping to zero) when optimizing for modern rewards like PAC-S, whereas DiCO maintains stability.

Methodology: Direct CLIP-Based Optimization

We propose DiCO, an "RL-free" alignment strategy inspired by Direct Preference Optimization (DPO). Instead of relying on unstable policy gradient methods, DiCO distills a reward model directly into the captioner's objective.

  1. Reward Distillation: The captioner generates multiple candidate captions for an image. An external evaluator (such as CLIP) is used to rank these candidates, identifying a "winner" and several "losers."
  2. Weighted Classification: The model is trained to increase the probability of the "winner" relative to the "losers" by solving a weighted classification problem. This allows the model to learn pairwise quality relations directly.
  3. KL-Regularization: To prevent the model from drifting into "hacked" territory, DiCO incorporates a per-token KL-penalty. This ensures the fine-tuned model stays close to its original pre-trained state, preserving linguistic fluency and preventing the generation of repetitive artifacts.
Overview of the DiCO training process. The captioner generates candidates, which are ranked by a CLIP-based evaluator to define a direct optimization objective based on quality distances.

Experimental Results

Experiments were conducted primarily on the COCO dataset using Karpathy splits, with further validation on benchmarks like nocaps, VizWiz, and TextCaps to ensure robustness across diverse domains.

Key Insights

  • Stability: DiCO demonstrates that direct optimization is significantly more stable than reinforcement learning. It avoids the "catastrophic forgetting" of linguistic rules that often plagues SCST when optimizing for semantic metrics.
  • The Role of Regularization: The parameter $\beta$ is crucial for balancing fluency and descriptiveness. A value of $\beta = 0.2$ was found to be the "sweet spot," allowing the model to optimize for target metrics while remaining grounded in its pre-trained knowledge.
  • Human Preference: In user studies, DiCO-generated captions were preferred over 50% of the time when compared to powerful baselines like BLIP-2 and SCST-tuned variants. Unlike SCST, which may collapse into repetitive n-grams, DiCO's preference-based learning encourages varied and detailed descriptions.

Conclusion

DiCO offers a robust alternative to traditional reinforcement learning for image captioning. By aligning models directly with high-quality evaluators, it produces captions that are more descriptive, fluent, and human-aligned. While the model can still exhibit factual errors or hallucinations in highly uncertain contexts, it represents a significant step forward in stable vision-language alignment. Future work will focus on improving factual grounding and entity recognition.

Summary updated

Abstract

The conventional training approach for image captioning involves pre-training a network using teacher forcing and subsequent fine-tuning with Self-Critical Sequence Training to maximize hand-crafted captioning metrics. However, when attempting to optimize modern and higher-quality metrics like CLIP-Score and PAC-Score, this training method often encounters instability and fails to acquire the genuine descriptive capabilities needed to produce fluent and informative captions. In this paper, we propose a new training paradigm termed Direct CLIP-Based Optimization (DiCO). Our approach jointly learns and optimizes a reward model that is distilled from a learnable captioning evaluator with high human correlation. This is done by solving a weighted classification problem directly inside the captioner. At the same time, DiCO prevents divergence from the original model, ensuring that fluency is maintained. DiCO not only exhibits improved stability and enhanced quality in the generated captions but also aligns more closely with human preferences compared to existing methods, especially in modern metrics. Additionally, it maintains competitive performance in traditional metrics. Our source code and trained models are publicly available at https://github.com/aimagelab/DiCO.
BibTeX
@inproceedings{moratelli2024revisiting,
  author = {Nicholas Moratelli and Davide Caffagni and Marcella Cornia and Lorenzo Baraldi and Rita Cucchiara},
  title = {{Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization}},
  booktitle = {BMVC},
  year = {2024},
}
Figure