L-CLIPScore: A Lightweight Embedding-Based Captioning Metric for Evaluating and Training

Li Li, Yingzhe Peng, Xu Yang, Ruoxi Cheng, Haiyang Xu, Ming Yan, Fei Huang

TL;DR

L-CLIPScore is a lightweight, embedding-based metric for image captioning that compresses the massive CLIP model by 3.4× and speeds up inference by 1.75× while maintaining high evaluation accuracy. By introducing a novel Similarity Regulator loss, the authors successfully distill vision-language alignment knowledge into a compact 99M parameter student model. Crucially, the paper demonstrates that this lightweight metric can act not only as an evaluator ("judge") but also as a training supervisor when combined with traditional n-gram metrics to prevent language collapse.

What Problem Does This Paper Address?

Evaluating image captioning models is notoriously tricky. Traditional metrics like BLEU and CIDEr rely on n-gram overlap, meaning they look for exact word matches between the generated caption and human reference captions. This approach penalizes perfectly valid paraphrases and rewards safe, repetitive, template-like phrasing.

To solve this, researchers recently turned to embedding-based metrics like CLIPScore, which use large pre-trained vision-language models to measure the semantic alignment directly between an image and a generated caption. While highly accurate and correlation-rich with human judgment, these models (such as CLIP ViT-B/32 with 338M parameters) are computationally heavy. This makes them impractical for quick inference, deployment on resource-constrained edge devices (like carry-on medical devices or tools for the visually impaired), or direct integration as a reinforcement learning reward during model training.

Overview of L-CLIPScore as a judge and supervisor

Methods

The authors introduce L-CLIP (Lightweight CLIP) and its corresponding metric, L-CLIPScore. They achieve dramatic efficiency gains through a combination of model compression and a novel distillation loss:

  1. Weight Multiplexing: To compress the encoder networks, the weights of a single learnable Transformer block are shared across multiple layers using learnable multiplex sub-networks. This compresses the vision encoder from 6 layers to 3 shared blocks, and the language encoder from 4 layers to 2 shared blocks.
  2. Matrix Decomposition: The massive word embedding matrix ($49408 \times 768$) is factorized into two smaller matrices ($49408 \times 256$ and $256 \times 768$), drastically cutting text encoder parameters.
  3. Similarity Regulator (SR) Loss: Standard contrastive distillation often fails to converge well on smaller datasets. To fix this, the authors designed the SR loss for multi-modal distillation. As illustrated below, it explicitly amplifies embedding similarity for matched image-text pairs and diminishes it for non-matched pairs, rapidly transferring the teacher's alignment capabilities to the student.
L-CLIP architecture and Similarity Regulator loss

L-CLIPScore can be deployed in two roles: as a Judge to evaluate caption quality (with or without reference text) and as a Supervisor to train captioning models via Self-Critical Sequence Training (SCST).

Datasets

  • Distillation & Pre-training: ImageNet-1M (1.2M images) and Conceptual Captions (3.3M captions) were used for single-modal distillation from the teacher encoders.
  • Training & Evaluation: MSCOCO (118,287 train / 5,000 val / 5,000 test pairs) served as the primary dataset for training the captioning models.
  • Human Correlation Benchmarks: Flickr8K-Expert, Flickr8k-CF, Composite dataset, Pascal-50S, and the FOIL dataset were utilized to measure how well the metric aligns with human judgment and resists adversarial noise.

Key Insights and Ideas

  • CLIP is Over-Parameterized for Evaluation: A model roughly one-third the size of the original CLIP can achieve identical, and sometimes superior, human correlation scores, proving that massive scale isn't strictly necessary for downstream evaluation tasks.
  • The Danger of Pure Embedding Rewards (Language Collapse): When a captioning model is trained solely using L-CLIPScore as a reinforcement learning reward, it suffers from "reward hacking." The model learns to output a chaotic string of highly correlated visual keywords (e.g., repeating "mirror mirror mirror") to maximize the embedding similarity score, completely breaking grammatical syntax.
  • Metric Complementarity: The authors discover that embedding-based metrics and n-gram metrics are perfectly complementary. Combining L-CLIPScore with CIDEr creates a balanced reward system: L-CLIPScore ensures rich visual-semantic grounding, while CIDEr acts as a syntactic regularizer to maintain proper grammar and sentence structure.

Key Results

  • Size and Speedups: L-CLIP reduces parameters from 338M to 99M (a 3.4× reduction) and cuts inference time for a 128-batch from 225ms to 124ms on an RTX 3090 GPU.
  • High Human Correlation: On the Pascal-50S dataset, reference-free L-CLIPScore achieved a mean accuracy of 81.4%, actually outperforming the original, uncompressed CLIPScore (80.7%).
  • Enhanced Training Performance: When used as a joint supervisor ($S_{Mix}$ with an optimal L-CLIPScore-to-CIDEr ratio of 7:3), the captioning model improved across the board, boosting the CIDEr score from 114.5 to 116.7 and the L-CLIPScore from 76.3 to 78.0 compared to baseline CIDEr training.

Limitations and Future Work

The primary limitation of L-CLIPScore—and embedding-based metrics in general—is evaluation bias. If a captioning model is trained using a specific embedding space (like CLIP), using that same embedding space to evaluate it will naturally yield artificially high scores.

To mitigate this single-model bias, the authors suggest that future work should focus on distilling multiple distinct, large-scale vision-language models (such as various vision-language BERTs) into a single, unified lightweight student model.

Summary updated

Abstract

We propose a novel embedding-based captioning metric termed as L-CLIPScore that can be used for efficiently evaluating caption quality and training captioning model. L-CLIPScore is calculated from a lightweight CLIP (L-CLIP), which is a dual-encoder architecture compressed and distilled from CLIP. To compress, we apply two powerful techniques which are weight multiplexing and matrix decomposition for reducing the parameters of encoders and word embedding matrix, respectively. To distill, we design a novel multi-modal Similarity Regulator (SR) loss to transfer more vision-language alignment knowledge. Specifically, SR loss amplifies the multi-modal embedding similarity if the given image-text pair is matched and diminishes the similarity if the pair is non-matched. By compressing and distilling by this novel SR loss, our L-CLIP achieves comparable multi-modal alignment ability to the original CLIP while it requires fewer computation resources and running time. We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality. We also discover that when using L-CLIPScore as the supervisor to train the captioning model, it should be mixed up by an n-gram-based metric and meanwhile analyze why using L-CLIPScore only will cause fail training.
BibTeX
@article{li2026l,
  author = {Li Li and Yingzhe Peng and Xu Yang and Ruoxi Cheng and Haiyang Xu and Ming Yan and Fei Huang},
  title = {{L-CLIPScore: A Lightweight Embedding-Based Captioning Metric for Evaluating and Training}},
  journal = {IEEE Trans. Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2026.3665006},
}

Loading related content...

Figure