TL;DR
L-CLIPScore is a lightweight, embedding-based metric for image captioning that compresses the massive CLIP model by 3.4× and speeds up inference by 1.75× while maintaining high evaluation accuracy. By introducing a novel Similarity Regulator loss, the authors successfully distill vision-language alignment knowledge into a compact 99M parameter student model. Crucially, the paper demonstrates that this lightweight metric can act not only as an evaluator ("judge") but also as a training supervisor when combined with traditional n-gram metrics to prevent language collapse.
What Problem Does This Paper Address?
Evaluating image captioning models is notoriously tricky. Traditional metrics like BLEU and CIDEr rely on n-gram overlap, meaning they look for exact word matches between the generated caption and human reference captions. This approach penalizes perfectly valid paraphrases and rewards safe, repetitive, template-like phrasing.
To solve this, researchers recently turned to embedding-based metrics like CLIPScore, which use large pre-trained vision-language models to measure the semantic alignment directly between an image and a generated caption. While highly accurate and correlation-rich with human judgment, these models (such as CLIP ViT-B/32 with 338M parameters) are computationally heavy. This makes them impractical for quick inference, deployment on resource-constrained edge devices (like carry-on medical devices or tools for the visually impaired), or direct integration as a reinforcement learning reward during model training.
Methods
The authors introduce L-CLIP (Lightweight CLIP) and its corresponding metric, L-CLIPScore. They achieve dramatic efficiency gains through a combination of model compression and a novel distillation loss:
- Weight Multiplexing: To compress the encoder networks, the weights of a single learnable Transformer block are shared across multiple layers using learnable multiplex sub-networks. This compresses the vision encoder from 6 layers to 3 shared blocks, and the language encoder from 4 layers to 2 shared blocks.
- Matrix Decomposition: The massive word embedding matrix ($49408 \times 768$) is factorized into two smaller matrices ($49408 \times 256$ and $256 \times 768$), drastically cutting text encoder parameters.
- Similarity Regulator (SR) Loss: Standard contrastive distillation often fails to converge well on smaller datasets. To fix this, the authors designed the SR loss for multi-modal distillation. As illustrated below, it explicitly amplifies embedding similarity for matched image-text pairs and diminishes it for non-matched pairs, rapidly transferring the teacher's alignment capabilities to the student.
L-CLIPScore can be deployed in two roles: as a Judge to evaluate caption quality (with or without reference text) and as a Supervisor to train captioning models via Self-Critical Sequence Training (SCST).
Datasets
- Distillation & Pre-training: ImageNet-1M (1.2M images) and Conceptual Captions (3.3M captions) were used for single-modal distillation from the teacher encoders.
- Training & Evaluation: MSCOCO (118,287 train / 5,000 val / 5,000 test pairs) served as the primary dataset for training the captioning models.
- Human Correlation Benchmarks: Flickr8K-Expert, Flickr8k-CF, Composite dataset, Pascal-50S, and the FOIL dataset were utilized to measure how well the metric aligns with human judgment and resists adversarial noise.
Key Insights and Ideas
- CLIP is Over-Parameterized for Evaluation: A model roughly one-third the size of the original CLIP can achieve identical, and sometimes superior, human correlation scores, proving that massive scale isn't strictly necessary for downstream evaluation tasks.
- The Danger of Pure Embedding Rewards (Language Collapse): When a captioning model is trained solely using L-CLIPScore as a reinforcement learning reward, it suffers from "reward hacking." The model learns to output a chaotic string of highly correlated visual keywords (e.g., repeating "mirror mirror mirror") to maximize the embedding similarity score, completely breaking grammatical syntax.
- Metric Complementarity: The authors discover that embedding-based metrics and n-gram metrics are perfectly complementary. Combining L-CLIPScore with CIDEr creates a balanced reward system: L-CLIPScore ensures rich visual-semantic grounding, while CIDEr acts as a syntactic regularizer to maintain proper grammar and sentence structure.
Key Results
- Size and Speedups: L-CLIP reduces parameters from 338M to 99M (a 3.4× reduction) and cuts inference time for a 128-batch from 225ms to 124ms on an RTX 3090 GPU.
- High Human Correlation: On the Pascal-50S dataset, reference-free L-CLIPScore achieved a mean accuracy of 81.4%, actually outperforming the original, uncompressed CLIPScore (80.7%).
- Enhanced Training Performance: When used as a joint supervisor ($S_{Mix}$ with an optimal L-CLIPScore-to-CIDEr ratio of 7:3), the captioning model improved across the board, boosting the CIDEr score from 114.5 to 116.7 and the L-CLIPScore from 76.3 to 78.0 compared to baseline CIDEr training.
Limitations and Future Work
The primary limitation of L-CLIPScore—and embedding-based metrics in general—is evaluation bias. If a captioning model is trained using a specific embedding space (like CLIP), using that same embedding space to evaluate it will naturally yield artificially high scores.
To mitigate this single-model bias, the authors suggest that future work should focus on distilling multiple distinct, large-scale vision-language models (such as various vision-language BERTs) into a single, unified lightweight student model.
Summary updated