TL;DR
Direct CLIP-Based Optimization (DiCO) is a novel training paradigm for image captioning that moves away from unstable reinforcement learning. By distilling rewards from high-quality evaluators like CLIP-Score directly into the model's objective, DiCO produces more descriptive, human-aligned captions while avoiding the "reward hacking" and model collapse common in traditional fine-tuning methods.
What Problem Does This Paper Address?
The standard way to train image captioners involves two steps: first, "teacher forcing" (predicting the next word based on ground-truth text), followed by fine-tuning with Self-Critical Sequence Training (SCST). SCST uses reinforcement learning to maximize specific metrics, traditionally n-gram overlap scores like CIDEr.
However, researchers are increasingly moving toward modern metrics like CLIP-Score and PAC-Score, which use multimodal embedding spaces to measure how well a caption actually matches an image. The problem is that SCST is notoriously unstable when optimizing these modern metrics. As shown in Figure 1, models often "hack" the reward—generating repetitive, ungrammatical gibberish (e.g., "a rusted rusty rusty...") that scores high on the metric but is useless to humans.
Methods
The authors propose DiCO, an "RL-free" alignment strategy inspired by Direct Preference Optimization (DPO) used in Large Language Models.
- Reward Distillation: Instead of using a separate reinforcement learning loop, DiCO distills a reward model directly into the captioner. It samples multiple candidate captions for an image and uses an external evaluator (like CLIP) to identify a "winner" and several "losers."
- Weighted Classification: The model is trained to increase the probability of the "winner" relative to the "losers" by solving a weighted classification problem. This mimics the pairwise quality relations found by the evaluator.
- KL-Regularization: To prevent the model from drifting into nonsensical "hacked" territory, DiCO includes a per-token KL-penalty. This ensures the fine-tuned model stays close to its original pre-trained state, maintaining linguistic fluency.
Datasets
The primary experiments were conducted on the COCO dataset using the standard Karpathy splits. To prove the method's robustness, the authors also evaluated generalization on six additional benchmarks:
- nocaps (out-of-domain objects)
- VizWiz (images taken by people who are blind)
- TextCaps (images with text)
- Conceptual Captions 3M (CC3M)
- FineCapEval (fine-grained descriptions)
- Flickr30k
Key Insights and Ideas
- Stability over Reinforcement: DiCO demonstrates that you don't need complex policy gradient methods to align a vision-language model. Direct optimization is significantly more stable and less prone to the "catastrophic forgetting" of linguistic rules.
- The Importance of $\beta$: The regularization parameter $\beta$ is crucial. A higher $\beta$ keeps the model grounded in its pre-trained knowledge, while a lower $\beta$ allows it to optimize more aggressively for the target metric. The authors found $\beta = 0.2$ to be the "sweet spot" for balancing fluency and descriptiveness.
- Avoiding the Repetition Trap: Unlike SCST, which often collapses into repetitive n-grams to maximize scores, DiCO's preference-based learning encourages more varied and detailed descriptions.
Key Results
DiCO achieves state-of-the-art performance across several dimensions:
- Modern Metrics: It significantly outperforms SCST and other CLIP-based optimization methods on PAC-S and CLIP-S.
- Human Preference: In a user study, captions generated by DiCO were preferred over 50% of the time when compared to powerful models like BLIP-2 and SCST-tuned variants.
- Qualitative Superiority: As seen in Figure 3, DiCO generates captions that are more descriptive and grammatically correct than SCST, which often produces repetitive errors when pushed to optimize for PAC-S.
Limitations and Future Work
Despite its strengths, the authors acknowledge several limitations:
- Factual Errors: The model can still make factual mistakes, such as misidentifying objects (e.g., calling balloons "kites").
- Entity Recognition: It may fail to recognize specific landmarks or brands (e.g., the Taj Mahal) if they were not prominent in the training data.
- Hallucination: When the subject of an image is highly uncertain, the model may rely on its learned priors and "hallucinate" details.
Future work could involve incorporating open-world knowledge or more robust factual grounding to address these descriptive inaccuracies.
Summary updated