Human-CLAP: Human-perception-based Contrastive Language-audio Pretraining

Taisei Takano, Yuki Okamoto, Yusuke Kanamori, Yuki Saito, Ryotaro Nagase, Hiroshi Saruwatari

This research paper, presented at APSIPA ASC 2025, introduces Human-CLAP, a method to align automated audio-text evaluation metrics with actual human perception. By fine-tuning a large-scale pretrained model on a small set of human-rated data, the authors significantly improved the reliability of "CLAPScore," a standard metric used in the growing field of text-to-audio generation.

TL;DR

Standard metrics for evaluating how well audio matches a text description (like CLAPScore) often fail to align with human judgment. This paper introduces Human-CLAP, a model fine-tuned on human subjective scores that improves the correlation between automated metrics and human perception by over 60%, using only a tiny fraction of the original training data.

What Problem Does This Paper Address?

As text-to-audio (TTA) models (like those that generate "the sound of a busy city street") become more advanced, we need objective ways to measure their quality. The current gold standard is CLAPScore, derived from Contrastive Language–Audio Pretraining (CLAP). CLAP maps audio and text into a shared mathematical space; the closer the two "embeddings" are, the higher the score.

However, the authors point out a major flaw: CLAP models are typically trained on binary data (this text matches this audio, or it doesn't). In reality, human perception is nuanced—a description might only partially match a sound, or the training data itself might contain noisy, incomplete labels. Consequently, the authors found that standard CLAPScore has a surprisingly low correlation with how humans actually rate audio-text relevance.

Methods

The authors propose Human-CLAP, which fine-tunes a pretrained LAION-CLAP model using human subjective scores as the ground truth. Their approach combines two distinct strategies:

  1. Regression Losses (MSE/MAE): The model is trained to directly predict the human score (rescaled to a 0–1 range) for a given audio-text pair.
  2. Weighted Symmetric Cross Entropy (wSCE): This is a novel twist on the standard contrastive loss. Instead of treating every pair in the dataset as a "perfect match," the loss function is weighted by the human subjective score. This teaches the model that some matches are "better" than others.

The final model uses a linear combination of these losses to refine the embedding space so that mathematical distance better reflects human-perceived relevance.

Datasets

  • RELATE Dataset: This is the core of the study, containing subjective relevance scores (on an 11-point scale) for audio-text pairs.
  • Audio Sources: The dataset includes natural audio from AudioCaps and synthesized audio from four state-of-the-art TTA models: AudioLDM, AudioLDM2, Tango, and Tango2.
  • Scale: The authors used approximately 2,383 pairs for training/validation and 2,405 pairs for testing. Notably, this training set is roughly 1/320th the size of the data used to train the original CLAP model.

Key Insights and Ideas

  • Nuance Matters: Standard CLAP assumes all training pairs are 100% matches. By introducing human scores, Human-CLAP acknowledges that relevance is a spectrum.
  • Efficiency of Human Feedback: You don't need millions of samples to align a model with human perception. A few thousand high-quality human ratings can effectively "steer" a massive pretrained model toward more human-like evaluation.
  • The "wSCE + MAE" Synergy: The best results came from combining the weighted contrastive loss with Mean Absolute Error. The authors found that while regression helps with score precision, the weighted contrastive loss is essential for correctly identifying and penalizing low-relevance (poorly matched) pairs.

Key Results

  • Significant Correlation Boost: The best Human-CLAP model achieved a Spearman’s rank correlation coefficient (SRCC) of 0.457, compared to just 0.280 for the baseline LAION-CLAP model—an improvement of over 0.17.
  • Consistency Across Sources: The improvement held true for both natural audio and synthesized audio from all four TTA generators.
  • Better Discriminator: Qualitative analysis (via scatter plots) showed that Human-CLAP is much more consistent at predicting scores across the entire range (0 to 10), whereas the baseline model often assigned high scores to pairs that humans rated poorly.

Limitations and Future Work

The authors acknowledge that their study is limited by the size and scope of the RELATE dataset. While the results are promising, the model's performance on "out-of-domain" data (audio types or descriptions not covered in the training set) remains to be seen.

Future work will involve collecting subjective scores for a wider variety of audio datasets to further generalize the Human-CLAP model and investigate its performance on more diverse, real-world audio-text relationships.

Summary updated

Abstract

Contrastive language-audio pretraining (CLAP) is widely used for audio generation and recognition tasks. For example, CLAPScore, which utilizes the similarity of CLAP embeddings, has been a major metric for the evaluation of the relevance between audio and text in text-to-audio. However, the relationship between CLAPScore and human subjective evaluation scores is still unclarified. We show that CLAPScore has a low correlation with human subjective evaluation scores. Additionally, we propose a human-perception-based CLAP called Human-CLAP by training a contrastive language-audio model using the subjective evaluation score. In our experiments, the results indicate that our Human-CLAP improved the Spearman's rank correlation coefficient (SRCC) between the CLAPScore and the subjective evaluation scores by more than 0.17 compared with the conventional CLAP.
BibTeX
@inproceedings{takano2025human,
  author = {Taisei Takano and Yuki Okamoto and Yusuke Kanamori and Yuki Saito and Ryotaro Nagase and Hiroshi Saruwatari},
  title = {{Human-CLAP: Human-perception-based Contrastive Language-audio Pretraining}},
  booktitle = {APSIPA},
  year = {2025},
  url = {https://openreview.net/forum?id=rYS6XVrf4P},
}

Loading related content...

Figure