Showing the Judge the Right Frames: Pairwise-Ranked Segment Refinement for Long-Video Soundtracks

Systems that write music for video increasingly use a large multimodal model as a critic, meaning a model that watches the video, listens to the music, and says whether they fit. These critics are unreliable. Frontier models agree with expert musicians well under 80% of the time, and on minute-long videos the critic cannot watch every frame, so it judges from a handful of stills. This proposal starts from two observations. First, a weak critic becomes much more useful when it is asked "which of these two is better?" many times and the answers are combined statistically, instead of being asked for a score. Second, the frames the critic sees should be chosen for each comparison: they should be the moments where the two candidate soundtracks actually differ in how well they fit the picture, not frames spread evenly or frames that are merely relevant to the scene.

We propose building a long-video soundtrack segment by segment. A generator proposes several candidates for each segment. A critic compares pairs of candidates while looking at a small, diverse set of frames picked to separate those two candidates. A Bayesian ranking model pools the noisy verdicts into a strength estimate and an uncertainty for each candidate. The loop then regenerates only the segment the model is least confident about, conditioning on the neighbouring segments it has already accepted. The claim to test is that, at a fixed budget of critic calls, choosing what the critic sees and how its verdicts are pooled improves human-judged soundtrack quality more than any of the simpler fixes: a stronger scalar scorer, more samples, or more frames.

Core Claim

In judge-guided refinement of video soundtracks, the critic's evidence is the bottleneck, not the generator. The evidence has two parts: which frames it is shown and how its verdicts are combined. Fixing both gives larger gains per critic call than any fix to the generator side.

Each ingredient has a precedent in a different setting. Correct Looks Better fits Bradley-Terry models (a model where each candidate has a hidden strength and the chance it wins a comparison depends on the strength difference) to pairwise LLM verdicts on text QA. Ranked this way, a judge with 4.9% accuracy on SimpleQA reached a Kendall distance of 20% from the true model ranking, compared with 38.2% for direct judging. Duel-Evolve runs that kind of ranking online inside an evolutionary loop, using Double Thompson Sampling (picking which pair to compare by drawing from the current uncertainty), but only over code and math text. TimeSearch-R uses a determinantal point process (DPP), a subset-selection rule that rewards high-quality items while penalising near-duplicates, to choose diverse frames. Its frames are chosen for a question being answered, not for a comparison being judged.

The new element is a pair-conditioned evidence set. Frames are selected to discriminate between two specific candidates, and the critic's verdict is treated as one noisy observation in a segment-level ranking model. To our knowledge, no work in the file selects frames on behalf of a critic, or ranks video-conditioned audio candidates by pairwise aggregation.

Research Program

The generator. We decompose a soundtrack for a video of several minutes into segments of roughly 10 seconds. That length matches what current video-to-music generators produce: VIBE generates fixed 10-second 48kHz clips and trains against scalar rewards from Qwen2.5-Omni. Our primary testbed is symbolic, built on V-MusProd, which generates chords, melody and accompaniment from video features and introduced the VMCP contrastive video-music retrieval score. We choose symbolic music for two reasons. Localized regeneration conditioned on neighbouring bars is natural there, and the SymMV dataset supplies a human-made soundtrack for every video, which we use as an anchor. We retrain V-MusProd on SymMV if weights are unavailable. A secondary audio testbed stitches clip-level outputs from a 10-second generator with crossfades, and its results are reported separately.

Evidence selection. For a comparison between candidates $a$ and $b$ on segment $k$, a cheap frame-level aligner gives scores $r_a(t)$ and $r_b(t)$ for each frame $t$ in the segment window plus a short margin. Here $r$ is a VMCP-style contrastive similarity combined with a motion-onset versus note-onset correlation. We build a DPP kernel whose quality term is $q_t = |r_a(t) - r_b(t)|$, how differently the two candidates fit frame $t$. Its similarity term is visual feature similarity between frames. We take the $m$ frames that maximise the determinant, with $m$ between 4 and 8.

This replaces the query-relevance quality term used by TimeSearch-R, BOLT's inverse-transform sampling, and AKS's split-and-judge with a discrimination term. Those methods supply the diversity machinery we reuse. The cheap aligner only decides where to look. It never decides the verdict, which keeps its known blindness to musical structure out of the reward.

Aggregation. Each candidate $c$ for segment $k$ has a latent strength $s_{k,c}$. Each critic verdict is modelled as

$$P(a \succ b \mid k, j) = \sigma\big(s_{k,a} - s_{k,b} + \gamma_j\,\delta_{ab}\big),$$

where $\sigma$ is the logistic function, $j$ indexes the critic, $\delta_{ab} \in \{-1, 1\}$ records which candidate was presented first, and $\gamma_j$ is a learned position bias for critic $j$.

Transitions are scored separately. The critic compares pairs of candidate boundaries between segments $k$ and $k+1$ on frames around the cut, and these verdicts feed analogous strengths over adjacent pairs. A Gaussian prior and Laplace posterior give an uncertainty for every strength. The approach follows the Thurstone MAP projection used in Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare, which turns pairwise verdicts into a continuous scale.

The loop. Hard checks run first and discard gross failures, in the spirit of VIBE's verifiable tempo and key rewards and LoopTTS's filter stage. Double Thompson Sampling then chooses which pairs to compare. When the posterior identifies a segment whose best candidate is either uncertain or below the anchor, the generator produces new candidates for that segment only, conditioned on the accepted neighbours.

The SymMV human soundtrack enters every segment's comparisons as a reference candidate. This addresses a known weakness of relative ranking: it cannot tell when every candidate is bad. If even the best generated candidate is reliably below the anchor, the loop knows to keep searching rather than to stop.

Our refinement differs from LoopTTS, which turns a single critic's deterministic diagnosis into edit instructions for a trained refiner on isolated speech. Here refinement means targeted resampling steered by pooled, uncertainty-aware verdicts. We run the critics as an ensemble of Qwen2.5-Omni and Gemini, and we also run CMI-RewardBench's CMI-RM as a pairwise judge on a music-only stream. CMI-RM is the strongest trained music judge in the file.

Baselines

The naive reference is a single pass of V-MusProd on SymMV with uniformly sampled visual conditioning and no critic at all. The strongest number reported on this benchmark is V-MusProd's VMCP P@5 of 15.7, up from 8.9, together with a 73% subjective preference over the prior baselines, both from Video Background Music Generation. Beating the naive pass means a significant human-preference margin over it, not merely a VMCP gain, because VMCP is the kind of correlation metric the proposal distrusts.

The strongest competitor is whole-track best-of-N with a scalar critic. We draw N full soundtracks and pick the one with the highest direct score, in one variant from Qwen2.5-Omni or Gemini prompted on uniformly sampled frames, and in another from CMI-RM's musicality score plus VMCP. This is VIBE's reward recipe applied at inference time. It is the strongest competitor because it needs no new machinery, and CMI-RM is a strong judge: 78.20% accuracy on CMI-Pref musicality against 65.80% for Gemini 3 Pro, and 73.43% on Music Arena CMI-RewardBench. The comparison is fair because both systems get exactly the same budget of generator calls and critic calls, measured in critic-model tokens, the same generator weights, and the same tuning budget for N, temperature and segment count, chosen on a held-out split. Beating it means a human pairwise preference of at least 60% over best-of-N at equal budget, plus a higher rate of matching or exceeding the human anchor.

We add Duel-Evolve applied to whole tracks: Bayesian Bradley-Terry ranking and Thompson sampling over complete soundtracks, with uniform frames shown to the critic. On text it reached 94.0% on MathBench and 37.4% on LiveCodeBench Duel-Evolve. No reported result found on Lacuna for any pairwise-ranked method on video-to-music.

Three ablations isolate the ingredients. The first keeps everything but shows the critic uniform frames instead of pair-discriminative DPP frames. The second keeps the frame selection but replaces Bradley-Terry pooling with raw win counts under round-robin pairing. The third keeps both but regenerates whole tracks instead of single segments. On FilmScoreDB we additionally report against C³GVS, which steers diffusion with scalar rhythm and emotion guidance and reports an F1 of 68.36. On ReelBench the relevant reference is VIBE's 65% human win rate over Video-Robin.

Why not just…?

Why not just use CMI-RM as the scorer and take best-of-N? This is the most serious threat, because a trained reward model beats frontier MLLMs on music. The answer has two parts. First, CMI-RM is conditioned on text, lyrics and audio prompts, not on video, and its musicality and alignment heads correlate at about 0.85, so it cannot isolate whether a candidate fits the picture. Second, whole-track selection discards good segments that sit inside bad tracks. LoopTTS found targeted correction preferred over regenerate-from-scratch 77% of the time. We do not argue around the threat: CMI-RM is included as a judge inside our loop. If whole-track best-of-N with CMI-RM matches our system under human evaluation, the conclusion is that segment-level search adds nothing, and the third ablation reports exactly this.

Why not just show the critic more frames, or evenly spaced ones? More frames cost critic tokens linearly. BOLT reports that 8 well-chosen frames beat denser uniform input for video QA, raising Video-MME from 53.8% to 56.1% on LLaVA-OneVision. Whether this transfers to judging audio-visual fit is untested, and KFS-Bench warns that downstream metrics confound frame quality with model noise. Our planted-defect experiment below measures this directly, with the frame budget swept, so a null result is cleanly visible.

Why not just ask the critic for a 1–10 score? Direct scoring roughly doubled ranking error compared with pairwise pooling on SimpleQA in Correct Looks Better. That evidence comes from text with ground-truth answers, so we re-test it in the planted-defect setting, where ground truth also exists.

Why not just optimize objective metrics? Spectral fidelity and FAD are blind to rhythm and structure, as A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset showed by giving unmetered noise FAD comparable to structured loops. SongEval found Audiobox-Aesthetic correlates only 0.61–0.66 with human song ratings. We report these metrics but do not optimize them.

Decisive Experiments

Planted-defect test. This is the cleanest check, and it separates the evidence question from generation entirely. On SymMV videos we take the human soundtrack and plant localized defects in one segment: a note-onset shift of 150–500ms against visible motion, a mood or key swap at a scene change, or a broken transition. We then ask each critic configuration to compare the clean and defective versions. The measure is detection accuracy and ranking Kendall distance as a function of frame budget $m$ and critic-call budget, crossing uniform, relevance-DPP and pair-discriminative-DPP frame selection with direct scoring, win counts and Bradley-Terry pooling. The proposal predicts that pair-discriminative frames with Bradley-Terry pooling reach a given detection accuracy with markedly fewer frames and calls. If uniform frames match it, the evidence-selection claim fails.

End-to-end test. On SymMV test videos, with FilmScoreDB excerpts as a second domain, all systems run at matched budgets. Expert listeners give blinded pairwise judgments of the system outputs and the human soundtrack, and these are fitted with Bradley-Terry, so that the evaluation itself does not use the critics being optimized. SongEval's five dimensions are rated on a subset of outputs. We also report VMCP, SongEval's automatic predictor, and the critic-assigned scores.

Reward hacking. We track the gap between critic-judged and human-judged gains over refinement iterations. A rising critic score with flat human preference is the signature of the judge being gamed. We expect the multi-critic ensemble and the human anchor to shrink this gap relative to single-critic best-of-N, and we report it either way.