Human-CLAP: Human-perception-based contrastive language-audio pretraining

Taisei Takano, Yuki Okamoto, Yusuke Kanamori, Yuki Saito, Ryotaro Nagase, Hiroshi Saruwatari

Abstract

Contrastive language-audio pretraining (CLAP) is widely used for audio generation and recognition tasks. For example, CLAPScore, which utilizes the similarity of CLAP embeddings, has been a major metric for the evaluation of the relevance between audio and text in text-to-audio. However, the relationship between CLAPScore and human subjective evaluation scores is still unclarified. We show that CLAPScore has a low correlation with human subjective evaluation scores. Additionally, we propose a human-perception-based CLAP called Human-CLAP by training a contrastive language-audio model using the subjective evaluation score. In our experiments, the results indicate that our Human-CLAP improved the Spearman's rank correlation coefficient (SRCC) between the CLAPScore and the subjective evaluation scores by more than 0.25 compared with the conventional CLAP.

Introduction

As text-to-audio (TTA) models—capable of generating complex soundscapes from simple text prompts—become increasingly sophisticated, the need for reliable automated evaluation metrics has grown. The current industry standard is CLAPScore, derived from Contrastive Language–Audio Pretraining (CLAP). This metric measures the mathematical distance between audio and text embeddings in a shared space.

However, standard CLAP models are typically trained on binary data (matching vs. non-matching pairs), which fails to capture the nuance of human perception. Human judgment of audio-text relevance is a spectrum, not a binary. This research addresses the low correlation between standard CLAPScore and human subjective ratings by introducing Human-CLAP, a model fine-tuned specifically on human perception data.

Methods

Human-CLAP fine-tunes a large-scale pretrained LAION-CLAP model using human subjective scores as the ground truth. The training process incorporates two primary strategies to align the embedding space with human judgment:

  1. Weighted Symmetric Cross Entropy (wSCE): A novel adaptation of the standard contrastive loss where each pair's contribution is weighted by its human subjective score. This teaches the model that some matches are more relevant than others.
  2. Regression Losses (MSE/MAE): The model is trained to directly predict human scores (rescaled to a 0–1 range) for given audio-text pairs.
Overview of the Human-CLAP training pipeline, aligning predicted similarity with human-scored similarity.

Datasets

The model was trained and validated using the RELATE Dataset, which contains 11-point subjective relevance scores for approximately 4,800 audio-text pairs. The dataset includes natural audio from AudioCaps and synthesized audio from four state-of-the-art TTA models: AudioLDM, AudioLDM2, Tango, and Tango2. Remarkably, the training set used for fine-tuning was roughly 1/320th the size of the original CLAP training data, demonstrating the high efficiency of human-centric fine-tuning.

Key Results

The experimental results demonstrate a significant boost in metric reliability. The best-performing Human-CLAP model achieved a Spearman’s rank correlation coefficient (SRCC) of 0.457, a substantial improvement over the 0.280 baseline of the standard LAION-CLAP model.

Correlation between CLAPScore and subjective scores for different loss functions across various audio sources.

Key insights from the results include:

  • Synergy of Losses: The combination of wSCE and Mean Absolute Error (MAE) yielded the best results. While regression helps with score precision, the weighted contrastive loss is essential for correctly penalizing low-relevance pairs.
  • Consistency: The improvement in correlation was consistent across all audio sources, including both natural recordings and various synthetic generators.
  • Improved Discrimination: Qualitative analysis shows that Human-CLAP is a much better discriminator, avoiding the common baseline error of assigning high scores to pairs that humans rated poorly.

Conclusion

Human-CLAP successfully bridges the gap between automated audio evaluation and human perception. By leveraging a small amount of high-quality subjective data, the model significantly improves the reliability of CLAPScore as an evaluation metric for text-to-audio generation. Future work will focus on expanding the diversity of the training data to ensure the model generalizes well across even broader audio-text domains.

Summary updated

Abstract

Contrastive language-audio pretraining (CLAP) is widely used for audio generation and recognition tasks. For example, CLAPScore, which utilizes the similarity of CLAP embeddings, has been a major metric for the evaluation of the relevance between audio and text in text-to-audio. However, the relationship between CLAPScore and human subjective evaluation scores is still unclarified. We show that CLAPScore has a low correlation with human subjective evaluation scores. Additionally, we propose a human-perception-based CLAP called Human-CLAP by training a contrastive language-audio model using the subjective evaluation score. In our experiments, the results indicate that our Human-CLAP improved the Spearman's rank correlation coefficient (SRCC) between the CLAPScore and the subjective evaluation scores by more than 0.25 compared with the conventional CLAP.
BibTeX
@inproceedings{takano2025human,
  author = {Taisei Takano and Yuki Okamoto and Yusuke Kanamori and Yuki Saito and Ryotaro Nagase and Hiroshi Saruwatari},
  title = {{Human-CLAP: Human-perception-based contrastive language-audio pretraining}},
  booktitle = {APSIPA},
  year = {2025},
  url = {https://openreview.net/forum?id=rYS6XVrf4P},
}
Figure