Introduction
Training effective image and video captioning models typically necessitates a large volume of human-annotated descriptions per visual input. Standard datasets like MSCOCO provide five captions per image to facilitate learning. However, high-quality captioning is resource-intensive. This paper investigates language resource efficiency: the ability of a model to maintain performance as the number of available captions per image decreases.
The authors identify a significant performance degradation in state-of-the-art models using Self-Critical (SC) reinforcement learning when moving from five captions to one. Mathematical analysis reveals that this decline is primarily due to increased noise in the reinforcement learning "baseline"—the reference used to evaluate generated captions—when fewer ground-truth references are available.
Methodology: Generalized Pairwise Comparison (GPC)
To address the instability of training with limited language resources, the authors propose Generalized Pairwise Comparison (GPC).
GPC Loss Function
In standard SC training, a sampled caption is compared against a single "greedy" baseline. GPC generalizes this by comparing the sampled caption against multiple reference captions. By averaging these comparisons, the variance in the baseline estimation is significantly reduced. While the error in standard SC scales with $1/K$ (where $K$ is the number of ground-truth captions), GPC utilizes $m$ comparison captions to stabilize the gradient estimation regardless of $K$.
Mixed Distribution Sampling
To provide a diverse set of captions for comparison, a "Mixed Distribution" strategy is employed. Captions are sampled from three distinct sources:
- Low-score: Random captions from the dataset (likely irrelevant).
- Medium-score: Captions sampled from the model’s current distribution.
- High-score: The caption generated via greedy decoding.
This mixture ensures the model encounters a full spectrum of quality, enhancing the informativeness of the pairwise comparisons.
Experimental Setup
The method was evaluated on two benchmarks:
- MSCOCO: Image captioning performance was tested in "synthesized" low-resource settings (1, 2, 3, or 4 captions per image).
- TGIF: A video captioning dataset (animated GIFs) that naturally contains only one caption per video, serving as a real-world low-resource test case.
Results and Discussion
GPC demonstrates superior language resource efficiency compared to standard Cross-Entropy (XE) and Self-Critical (SC) approaches.
On MSCOCO, GPC achieved a CIDEr score of 96.2 in the single-caption (re-1) setting, significantly outperforming the standard SC model's score of 88.4. Notably, GPC using only 2.5 captions per image on average matched the performance of SC using the full 5 captions. On the TGIF dataset, GPC improved the CIDEr score from 49.8 to 52.1, setting a new state-of-the-art for this low-resource task.
Conclusion
The performance drop in reinforcement learning for captioning is fundamentally a variance problem. By decoupling the number of human annotations from the number of training comparisons, GPC provides a robust framework for learning from sparse data. Future work will focus on improving reward estimation accuracy when ground-truth references are limited and automating the definition of score distribution thresholds.
Summary updated