L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training

Li Li, Yingzhe Peng, Xu Yang, Ruoxi Cheng, Haiyang Xu, Ming Yan, Fei Huang

Abstract

We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.

Introduction

Evaluating the quality of image captions is a fundamental challenge in computer vision and natural language processing. Traditional metrics such as BLEU, METEOR, and CIDEr rely on n-gram overlap between generated captions and ground-truth references. While computationally efficient, these metrics often fail to capture semantic nuances; a caption that is semantically accurate but uses different vocabulary than the reference may receive a disproportionately low score.

Overview of L-CLIPScore: (a) Ground truth image-caption pair, (b) Comparison of L-CLIPScore and CIDEr, (c) Distillation process, and (d) Training application.

To address these limitations, embedding-based metrics like CLIPScore have emerged, utilizing the shared vision-language space of the CLIP model to measure similarity. However, the original CLIP models are computationally heavy, making them expensive for large-scale evaluation and difficult to integrate directly into training loops as reward functions. Furthermore, training models solely on embedding similarity often leads to "reward hacking," where models generate repetitive keywords to maximize similarity at the expense of grammatical structure. This paper introduces L-CLIPScore, a lightweight alternative that maintains high performance while significantly reducing computational overhead.

Methodology

The development of L-CLIPScore involves a sophisticated compression and distillation pipeline designed to retain the semantic alignment of the original CLIP model within a much smaller footprint.

Model Compression

The authors employ two primary techniques to reduce the parameter count of the CLIP encoders:

  1. Weight Multiplexing: This technique shares weights across different Transformer layers. By utilizing small, learnable sub-networks to modulate shared weights, the model can maintain expressive power while drastically reducing the total number of unique parameters.
  2. Matrix Decomposition: The large word embedding matrix is decomposed into smaller factors. This significantly reduces the memory footprint of the language encoder without sacrificing the richness of the vocabulary representation.

Together, these methods shrink the model from 338M parameters to 99M, a 3.4x reduction.

Similarity Regulator (SR) Loss

To bridge the performance gap between the large teacher (CLIP) and the small student (L-CLIP), a novel Similarity Regulator (SR) loss is introduced. Unlike standard distillation that simply encourages the student to mimic the teacher's output embeddings, the SR loss explicitly "amplifies" the similarity for matched image-text pairs and "diminishes" it for non-matched pairs. This approach makes the student model more discriminative, allowing it to achieve alignment capabilities comparable to the original CLIP even when trained on smaller datasets.

The architecture of LCLIP featuring weight multiplexing and matrix decomposition, alongside the Similarity Regulator (SR) loss mechanism.

Mixed Reward Training

A key contribution of this work is the exploration of L-CLIPScore as a training supervisor. To prevent the "language collapse" typically seen when training on embedding similarities alone, the authors propose a Mixed Reward ($S_{Mix}$). This reward combines the semantic grounding of L-CLIPScore with the syntactic constraints of CIDEr. This hybrid approach ensures that the model generates captions that are both visually accurate and linguistically fluent.

Experiments and Results

The effectiveness of L-CLIPScore was evaluated across multiple dimensions, including efficiency, human correlation, and its utility in training.

Efficiency

L-CLIPScore demonstrates significant practical advantages. It is 3.4x smaller than the original CLIP-based metric and nearly 2x faster in terms of inference speed (8ms vs. 14ms per pair). This makes it highly suitable for real-time evaluation and large-scale training.

Human Correlation

On the Pascal-50S dataset, L-CLIPScore achieved a mean accuracy of 81.4%, slightly outperforming the original CLIPScore (80.7%). This indicates that the compression and distillation process successfully preserved—and in some cases refined—the model's ability to align with human judgment.

Training Performance

Models trained with the mixed reward ($S_{Mix}$) showed superior performance compared to those trained with CIDEr alone. The authors found that a balanced ratio (e.g., 7:3 CIDEr to L-CLIPScore) provided the best results, effectively grounding the captions in the image content while maintaining high-quality language generation.

Conclusion

L-CLIPScore represents a significant step forward in image captioning evaluation. By combining weight multiplexing, matrix decomposition, and the novel SR loss, the authors created a metric that is both lightweight and highly effective. The research highlights the complementary nature of n-gram and embedding-based metrics, suggesting that the future of captioning lies in hybrid approaches that balance syntax and semantics. Furthermore, the analysis of "reward hacking" provides valuable insights for researchers looking to use vision-language models as training supervisors.

Summary updated

Abstract

We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.
BibTeX
@article{li2026l,
  author = {Li Li and Yingzhe Peng and Xu Yang and Ruoxi Cheng and Haiyang Xu and Ming Yan and Fei Huang},
  title = {{L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training}},
  journal = {IEEE Trans. Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2026.3665006},
}
Figure