Large music and audio archives are found almost entirely through the text people attach to them: titles, artist names and a few genre tags. When those descriptions are thin, a recording is effectively invisible no matter what it sounds like. The obvious fix is to have a model listen to each file and write richer tags. There is a catch the audio-evaluation literature knows well and the retrieval literature mostly ignores. The automatic quality numbers such a system would record, such as fidelity scores and embedding-distribution distances, often disagree with what trained listeners hear. A pristine recording of incoherent notes scores well, and a rough demo with strong song structure scores badly. An index built on those numbers would mis-describe exactly the long-tail material that most needs describing.
This note proposes treating that disagreement as information to store, not as noise to be averaged away. Each track gets two kinds of quality fields. Signal-level descriptors come from deterministic measurement tools. Perceptual descriptors come from predictors calibrated on expert listener ratings. Every field records which source it came from. A narrow arbitration step, invoked only when the two kinds of field disagree, decides whether the disagreement reflects a defect or an intended style (lo-fi, tape saturation, deliberately loose timing) and writes that decision as its own searchable field. The claim is that this separation of provenance, rather than a richer single caption or a larger debating panel, is what lets a search system answer queries like "rough-sounding demo with a strong chorus structure", which current text-to-music retrieval cannot express.
Core Claim
Discovery metadata for audio should separate three things: what was measured from the signal, what calibrated listeners' proxies predict, and a context judgment that reconciles them. The proposal predicts that retrieval gains will concentrate on tracks where objective and perceptual quality diverge most. On the remaining tracks, a simpler pipeline that just appends a perceptual score should do about as well. If the gain does not concentrate on the divergent tracks, the proposal's central mechanism is wrong.
What the evidence supports, and where the gap is
Three findings motivate the design. Towards More Usable Dataset Search studied 1,947 real dataset-search queries. It found that 96% use metadata elements, 64% require specific content values and 50% reference schema concepts. Searchers want structured, filterable facts about content, not prose alone. SongEval collected expert ratings on 2,399 full-length generated songs across five perceptual dimensions. It reports that existing aesthetic metrics such as Audiobox-Aesthetic correlate with human averages at only 0.61–0.66 Pearson, while a UTMOS-style predictor trained on its ratings reaches 0.912. A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset shows that distribution metrics reward drum timbre even when the rhythm is unmetered. Improving Automatic Jazz Melody Generation by Transfer Learning Techniques shows experts preferring the models that statistical metrics rank lower. Taken together, any "quality" facet derived from objective metrics will be wrong in a systematic and predictable direction.
A fourth finding constrains the design. CMI-RewardBench shows that musicality judgments depend on context, since low fidelity is a feature when "lo-fi" is requested. It also shows that its reward model's musicality and instruction-alignment heads correlate at about 0.85 Spearman, meaning a single scorer blends "is this good" with "is this what was asked for." We take this as a reason to keep the context decision out of the quality score itself and to store it separately.
Three systems sit closest to this proposal. Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval (TTMR++) expands existing tags into pseudo-captions with a fine-tuned language model and trains an audio–text contrastive retriever. Its caption synthesis does not listen to the audio, and the authors report weakness on tempo, key and rhythm. Rethinking Music Captioning with Music Metadata LLMs does listen. It predicts a fixed JSON schema (genre, mood, instrumentation, tempo) and then stylizes it with a frozen text model, matching end-to-end captioning at roughly 0.625 versus 0.636 SBERT similarity. Its schema has no perceptual-quality fields and no notion of where a field came from. We reuse its decoupling of structured extraction from prose generation. MetaSynth decomposes metadata writing into retriever, generator, evaluator-panel and rewrite agents for text product catalogs, driven by click feedback. We borrow the idea of separating generation from verification, but our verification target is acoustic, not promotional. Two further systems use separate judging roles on outputs: Multi-Agent-as-Judge (MAJ-EVAL) uses persona-based debating judges on text, and Diagnose, Then Refine (LoopTTS) uses a cheap metric filter followed by an audio-language-model judge for speech prosody. Neither produces index metadata. LoopTTS's filter-then-judge pattern, which pays for the expensive judge only after a cheap screen, is the cost structure we adopt.
Research Program
The system has four roles, and the split follows where each kind of knowledge comes from, not a wish for many agents. A measurement role runs deterministic music-analysis tools for tempo, key, loudness, section boundaries and duration, along with the objective quality metrics already in use. A listening role runs perceptual predictors: a SongEval-style predictor for coherence, memorability, vocal naturalness, structural clarity and musicality, and the CMI-RM reward model for context-conditioned musicality. For each perceptual dimension $k$ we compute a divergence score
$$d_k = z(o_k) - z(p_k),$$
where $o_k$ is the objective metric most often used as a proxy for that dimension, $p_k$ is the calibrated perceptual prediction, and $z(\cdot)$ standardizes each score across the archive. Large $|d_k|$ marks a track where the two sources disagree.
An arbitration role runs only when $|d_k|$ exceeds a threshold. It is an audio-capable language model that receives the audio, both sets of scores and any existing tags. It returns a constrained decision: defect, intended style or uncertain, plus a short justification. These decisions are stored as separate fields, each tagged with its source.
A snippet role then works at query time. A frozen text model turns the stored fields into a query-conditioned description of the kind Chen et al. found users prefer. It never invents values, and it may only verbalize stored fields.
The arbitration step is deliberately narrow because the evidence does not support broad trust in audio-language-model judgment. Gemini 3 Pro reaches only 65.80% on CMI-Pref musicality, so the arbiter is never asked to score quality, only to classify a disagreement already detected by calibrated tools. The analysis component asks three questions. Where in an archive do divergences concentrate (by genre, by generator, by recording condition)? How often does the arbiter's style-versus-defect call match expert preference? How much retrieval quality does each stored field contribute?
Baselines
The naive reference is retrieval over creator-supplied text alone, namely the titles and tags shipped with each track, using standard text matching. It measures how invisible the archive is before any enrichment.
The strongest existing listen-and-describe method is the Metadata LLM pipeline of Rethinking Music Captioning with Music Metadata LLMs. It is the closest published system that actually processes audio into structured fields, and it reports about 0.625 SBERT on MusicCaps. We will run it with its published schema on the same audio and the same downstream retriever.
The strongest dense-retrieval competitor is TTMR++ from Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval. It reports Recall@10 of 29.2 on MusicCaps (up from 13.6) and artist-similarity nDCG@200 of 0.187 on DimSim. It bypasses metadata altogether by embedding audio directly, so it tests whether structured fields are needed at all. We will use released weights where available. Otherwise we will retrain on the same public data under a matched GPU budget, and we will also evaluate a hybrid that adds our fields as filters on top of its ranking.
The most dangerous baseline is the bolt-on pipeline. It takes the Metadata LLM schema and appends the raw SongEval-predictor and CMI-RM scores as extra numeric fields, with no divergence computation and no arbitration. It isolates whether anything beyond "add a good perceptual score" matters.
Two ablations isolate the key ingredient. The first replaces the arbiter with a fixed rule: mark as style if any tag contains a style keyword such as lo-fi, else defect. The second removes provenance tags so that the retriever sees merged quality fields.
All systems share the same audio, the same retriever, the same query set and an equal hyperparameter-tuning budget on a common development split. Language-model calls are capped per track at the cost of the proposed system. Beating the bolt-on pipeline means a statistically significant gain in nDCG@10 on the high-divergence slice of the benchmark, with no significant loss elsewhere. On MusicCaps, our system should stay within noise of TTMR++'s reported Recall@10 of 29.2 when used as a hybrid.
Why not just…?
Why not just have a frontier audio-language model listen and write everything in one pass? The direct evidence is CMI-RewardBench. There, Gemini 3 Pro agrees with experts on musicality only 65.80% of the time, against 78.20% for the trained CMI-RM. A single-pass writer would therefore put its least reliable judgment, overall quality, straight into the index. We include this baseline anyway, with a prompt that asks for the same fields. If it matches our system on the divergence slice, the arbitration design is unnecessary.
Why not just append a calibrated perceptual score and skip the divergence and arbitration machinery? This is the real threat, and we take it seriously. SongEval's 0.912 Pearson predictor may already capture most of what listeners mean, and CMI-RM already conditions on context. The proposal survives only if two things hold. First, the arbiter's style-versus-defect field must add retrieval value on context-dependent queries. Second, separating sources must help users filter on queries where fidelity and musicality point in opposite directions. The bolt-on baseline and the rule-based ablation settle this directly. If bolt-on ties, we will report that calibrated scores suffice and that decomposition adds cost without benefit. That would itself be a useful constraint for the multi-agent metadata line represented by MetaSynth.
Why not just skip metadata and use dense audio–text retrieval? TTMR++ shows dense retrieval works well for genre and mood-style queries. However, its authors report weakness on tempo, key and rhythmic properties, and Chen et al. show that most real queries carry structured content constraints. Dense embeddings also cannot express "low fidelity but strong structure" without having been trained on such contrasts. The comparison on structured-constraint queries, and the hybrid configuration, will show whether fields add value beyond the embedding. We expect the hybrid to win, not the fields alone.
Why not just run a MAJ-EVAL-style persona debate over every track? MAJ-EVAL raised Spearman correlation from 0.31 for G-Eval to 0.47 on StorySparkQA, but at about $0.42 per item, and on text only. Archive indexing touches every track, so we confine the expensive step to the divergent minority and measure what fraction of tracks triggers it.
Proposed Experiments
The main benchmark is built on SongEval. Its full-length songs and five expert-rated dimensions provide ground truth for perceptual constraints, and its 3.5-minute average length tests long-form structure. Because the perceptual predictor is trained on SongEval, we use five-fold cross-fitting, so that every song's predicted scores come from a model that never saw it.
Queries are written following Chen et al.'s characterization: a topic or genre, a content value such as tempo range or instrumentation, and a schema-level concept. A perceptual constraint is then added, phrased in both directions (for example "polished but aimless" and "rough but well-built"). Relevance is judged by musically trained annotators on pooled top-ranked results from all systems. Perceptual constraints are scored against SongEval's human ratings, never against any system's predictions. Reported metrics are nDCG@10 and Recall@10 overall and on the top-quintile divergence slice, defined by $|d_k|$ computed with human ratings in place of predictions. There is no reported result on Lacuna for any system on such a query set, so all numbers here are new.
The arbitration step is evaluated separately on CMI-Pref. Pairs with style-laden instructions are recast as defect-versus-style decisions, and agreement with the 31 experts' preferences is measured against CMI-RM's own preference as the reference point (78.20% on musicality).
Regression checks use MusicCaps, Song Describer and DimSim through the hybrid retriever. They should not fall below the TTMR++ figures above.
User-facing snippets are rated on a 5-point scale by annotators. Chen et al.'s 3.04 for illustrative and 1.91 for query-biased snippets serve only as an anchor from a different domain. We also measure a hallucination rate: the fraction of snippet claims not supported by a stored field.
The decisive checks are three. First, the gain over the bolt-on baseline should concentrate in the divergence slice and fade outside it. Second, the arbiter should beat the keyword rule on CMI-Pref style decisions. Third, arbitration should be triggered on a minority of tracks, keeping cost per track far below a full debate. If the first check fails, the core claim fails.
References
- Towards More Usable Dataset Search: From Query Characterization to Snippet Generation
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- Enriching Music Descriptions with A Finetuned-LLM and Metadata for Text-to-Music Retrieval
- Rethinking Music Captioning with Music Metadata LLMs
- MetaSynth: Multi-Agent Metadata Generation from Implicit Feedback in Black-Box Systems
- Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction