Abstract
Finding the ideal sound effect is a critical yet time-consuming task in professional audio production. Traditional keyword-based search systems often fail due to the "annotation gap" and rigid metadata requirements. This paper presents CLAP-UI, an AI-assisted sound search system utilizing Contrastive Language-Audio Pre-training (CLAP) to enable semantic, natural language queries. Through a user study with 20 audio professionals, we demonstrate that CLAP-UI significantly reduces user frustration and improves search efficiency compared to traditional keyword-matching libraries like the BBC Sound Effect Library.
Introduction
In professional audio production—spanning film, radio, and gaming—locating the "perfect" sound effect (SFX) is a notorious bottleneck. Traditional sound libraries rely on manual human annotations, which create two major issues: an annotation gap, where labels are incomplete or overly specific, and keyword rigidity, where simple word-matching fails if a query does not exactly match the metadata. This discourages users from using descriptive, natural language. CLAP-UI aims to bridge this gap by moving from word-matching to semantic-matching, allowing creators to describe the context or "vibe" of a sound.
Methods: The CLAP-UI System
The core of the system is CLAP (Contrastive Language-Audio Pre-training). The model uses a dual-encoder setup—an audio encoder (HTSAT) and a text encoder (RoBERTa)—to map both modalities into a shared high-dimensional embedding space. During training, the model learns to align matching audio-text pairs.
The researchers developed a web-based interface, CLAP-UI, which supports three primary interaction types:
- Natural Language Search: Descriptive sentence queries.
- Audio-to-Audio Search: Uploading a sound to find similar ones.
- Iterative Refinement: A "Search Similar Sound" button on every result.
The backend was optimized using Flask and Javascript, achieving sub-0.5 second response times across a database of nearly 2 million files from AudioSet.
Evaluation
The system was evaluated against the BBC Sound Effect Library, a professional standard containing over 33,000 manually annotated clips. Twenty professional participants were tasked with finding sounds for 12 specific scripts using both systems.
Key Insights and Results
CLAP-UI demonstrated superior semantic intelligence. For instance, for a prompt like "a man starts shouting excitedly," the traditional BBC system returned horse noises due to keyword mismatches, while CLAP-UI correctly identified human vocalizations. Participants noted that the AI acted as a creative partner, allowing them to search for "imaginary" sounds or conceptual atmospheres.
Quantitatively, CLAP-UI was rated significantly easier to use ($p = 1.14 \times 10^{-8}$). On a 0-10 difficulty scale, CLAP-UI's scores were heavily skewed toward "Very Easy," while the BBC system's scores were more evenly distributed.
Workload metrics showed a statistically significant improvement in perceived performance ($p = 0.010$) and a significant reduction in frustration ($p = 0.026$). Participants gave the system an average score of 7.65/10 for their intention to use it in future workflows.
Discussion and Future Work
While the AI was better at finding sounds, users missed the text descriptions found in traditional libraries—a phenomenon termed the "Description Paradox." In the BBC system, users could quickly read a label to "discount" a sound; in CLAP-UI, the lack of metadata forced users to listen to more clips. Future work will integrate AI-generated audio captioning to solve this. Additionally, while the system is dataset-agnostic, the use of AudioSet (sourced from YouTube) sometimes resulted in lower audio fidelity than professional libraries. Future iterations will focus on indexing higher-fidelity sources and improving the model's handling of complex multi-sound queries.