Abstract
We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.
Introduction
Evaluating the quality of image captions is a fundamental challenge in computer vision and natural language processing. Traditional metrics such as BLEU, METEOR, and CIDEr rely on n-gram overlap between generated captions and ground-truth references. While computationally efficient, these metrics often fail to capture semantic nuances; a caption that is semantically accurate but uses different vocabulary than the reference may receive a disproportionately low score.
To address these limitations, embedding-based metrics like CLIPScore have emerged, utilizing the shared vision-language space of the CLIP model to measure similarity. However, the original CLIP models are computationally heavy, making them expensive for large-scale evaluation and difficult to integrate directly into training loops as reward functions. Furthermore, training models solely on embedding similarity often leads to "reward hacking," where models generate repetitive keywords to maximize similarity at the expense of grammatical structure. This paper introduces L-CLIPScore, a lightweight alternative that maintains high performance while significantly reducing computational overhead.
Methodology
The development of L-CLIPScore involves a sophisticated compression and distillation pipeline designed to retain the semantic alignment of the original CLIP model within a much smaller footprint.
Model Compression
The authors employ two primary techniques to reduce the parameter count of the CLIP encoders:
- Weight Multiplexing: This technique shares weights across different Transformer layers. By utilizing small, learnable sub-networks to modulate shared weights, the model can maintain expressive power while drastically reducing the total number of unique parameters.
- Matrix Decomposition: The large word embedding matrix is decomposed into smaller factors. This significantly reduces the memory footprint of the language encoder without sacrificing the richness of the vocabulary representation.
Together, these methods shrink the model from 338M parameters to 99M, a 3.4x reduction.
Similarity Regulator (SR) Loss
To bridge the performance gap between the large teacher (CLIP) and the small student (L-CLIP), a novel Similarity Regulator (SR) loss is introduced. Unlike standard distillation that simply encourages the student to mimic the teacher's output embeddings, the SR loss explicitly "amplifies" the similarity for matched image-text pairs and "diminishes" it for non-matched pairs. This approach makes the student model more discriminative, allowing it to achieve alignment capabilities comparable to the original CLIP even when trained on smaller datasets.
Mixed Reward Training
A key contribution of this work is the exploration of L-CLIPScore as a training supervisor. To prevent the "language collapse" typically seen when training on embedding similarities alone, the authors propose a Mixed Reward ($S_{Mix}$). This reward combines the semantic grounding of L-CLIPScore with the syntactic constraints of CIDEr. This hybrid approach ensures that the model generates captions that are both visually accurate and linguistically fluent.
Experiments and Results
The effectiveness of L-CLIPScore was evaluated across multiple dimensions, including efficiency, human correlation, and its utility in training.
Efficiency
L-CLIPScore demonstrates significant practical advantages. It is 3.4x smaller than the original CLIP-based metric and nearly 2x faster in terms of inference speed (8ms vs. 14ms per pair). This makes it highly suitable for real-time evaluation and large-scale training.
Human Correlation
On the Pascal-50S dataset, L-CLIPScore achieved a mean accuracy of 81.4%, slightly outperforming the original CLIPScore (80.7%). This indicates that the compression and distillation process successfully preserved—and in some cases refined—the model's ability to align with human judgment.
Training Performance
Models trained with the mixed reward ($S_{Mix}$) showed superior performance compared to those trained with CIDEr alone. The authors found that a balanced ratio (e.g., 7:3 CIDEr to L-CLIPScore) provided the best results, effectively grounding the captions in the image content while maintaining high-quality language generation.
Conclusion
L-CLIPScore represents a significant step forward in image captioning evaluation. By combining weight multiplexing, matrix decomposition, and the novel SR loss, the authors created a metric that is both lightweight and highly effective. The research highlights the complementary nature of n-gram and embedding-based metrics, suggesting that the future of captioning lies in hybrid approaches that balance syntax and semantics. Furthermore, the analysis of "reward hacking" provides valuable insights for researchers looking to use vision-language models as training supervisors.
Summary updated