When Pluralistic Curation Stalls: Saddle Points in Recursive Retraining of Music Generators

Generative models are increasingly retrained on their own outputs after those outputs have been filtered by human or automated judges. On a community platform, this is how a music generator would come to learn what a scene "sounds like." A recent theoretical result says the loop does not have to collapse into one bland style. If the filter combines several competing judges, each standing for a different community taste, the retrained model provably converges to a compromise that keeps every taste represented. That proof assumes each retraining round fits the curated data exactly. Real models are trained by a fixed number of gradient steps per round. We argue that this gap matters for a specific reason. Early in training, a generator that blends all tastes into one "average" style sits at a saddle point, a configuration where the gradient is zero but some directions still lead downhill. The more similar the judges are, the flatter the downhill direction. With a finite step budget the model may never leave that point, and each round of curation then reinforces the average.

The proposal tests this mechanism on music. Music is a domain where the usual automatic metrics cannot see the failure: an audio-texture score such as Fréchet Audio Distance can stay flat while the music collapses stylistically. The motivation comes from work on online identity and community norms. Those papers worry that algorithmic curation homogenizes the cultural material people form identities around. Our contribution is not a new optimizer. It is a measurable account of when the pluralistic guarantee survives practical training, a cheap diagnostic for when it does not, and a test of whether the cheapest known saddle-escape fixes are enough.

Core Claim

Curated Synthetic Data Doesn't Have to Collapse studies recursive retraining where synthetic samples are kept with probability set by $M$ reward functions under a Bradley-Terry rule, a standard model of pairwise preference. It proves the resulting distribution converges to a weighted Nash bargaining solution, and that diversity survives through a phase transition governed by a "leakage gap" between the reward basins. On CIFAR-10, moving from one reward to five lowered FID from about 100 to about 36. Everything rests on an exact maximum-likelihood fit per round.

Our claim is that, under a per-round budget of $T$ gradient steps, the outcome is set by the product of $T$ and the negative curvature at a symmetric "compromise" configuration, not by the leakage gap alone. Here the symmetric configuration means one where the generator's modes, or its internal style components, coincide.

The reasoning is the linearized dynamics near a strict saddle. A small deviation $\delta_0$ along the unstable direction grows like $(1+\eta|\lambda_{\min}|)^t$. Here $\eta$ is the step size and $\lambda_{\min}<0$ is the most negative eigenvalue of the training loss's Hessian (its matrix of second derivatives) at the compromise point. Leaving a neighborhood of radius $r$ therefore takes roughly

$$ t_{\text{esc}} \approx \frac{\log(r/\delta_0)}{\eta\,|\lambda_{\min}|}. $$

We hypothesize that $|\lambda_{\min}|$ shrinks as the judges become more alike, that is, as the leakage gap closes. When $t_{\text{esc}} > T$, each round ends near the compromise point. The curated data from that round is then drawn mostly from the averaged style, and the next round's starting point moves closer to the stable side. Nothing in the idealized proof can see this compounding of plateaus. It predicts collapse in exactly the regime where the theory predicts diversity.

The musical setting matters for two reasons. First, judges built from existing music reward models are strongly entangled. CMI-RewardBench reports a Spearman correlation of about 0.85 between its reward model's musicality and instruction-alignment heads. "Pluralistic" panels assembled from such heads may sit in the small-curvature regime by construction. Second, the FreeSound Loop benchmarking study shows that distribution-matching scores reward correct acoustic texture while ignoring rhythmic structure. The collapse we predict would therefore be invisible to Fréchet Audio Distance and visible only to perceptual judges and listeners.

Research Program

The work has three layers, each feeding the next.

The first layer is an analyzable case. We reproduce the Gaussian-mixture setting of the pluralistic-curation paper with a finite-capacity mixture generator trained by gradient descent. Here $\lambda_{\min}$ at the symmetric point can be computed in closed form as a function of the reward geometry. We then check two things across a grid of leakage gaps and step budgets. The first is whether the escape-time relation above predicts which runs keep diversity. The second is whether the phase boundary in the $(T, \text{gap})$ plane departs from the infinite-capacity boundary in the predicted direction.

The second layer is a diagnostic usable on deep generators. Following Agarwal et al., who extract negative-curvature directions in linear time using Hessian-vector products, we estimate $\lambda_{\min}$ at the end of each retraining round by power iteration. We log it alongside the spread of per-judge rewards across samples. A stall would appear as a plateau in that spread at the same time as a small negative $\lambda_{\min}$.

We expect a complication. The pluralistic-curation paper bypasses the loss landscape entirely, and the open-problem notes on curvature-exploitation methods point out that overparameterized networks have highly degenerate Hessians. We therefore report the minimum eigenvalue restricted to the subspace orthogonal to near-zero directions, and we treat the diagnostic as empirical rather than as a proof-backed quantity.

The third layer is the music testbed. The generator is a StyleGAN2 adapted to mel-spectrograms of one-bar drum loops, the configuration Hung et al. found competitive with audio-specific models, trained on Looperman. Each retraining round curates its synthetic loops with a panel of $M$ judges. We build two kinds of panel. Deliberately entangled panels use the CMI-RM musicality and alignment heads, plus heads of the SongEval dimension predictor, which reaches Pearson 0.912 with expert ratings on full songs. Deliberately separated panels use judges fine-tuned on disjoint style clusters of the loop data. One honest caveat: the SongEval predictor was trained on full-length songs, and its transfer to two-second loops must itself be validated against a small listening set before its scores are trusted.

The community-curation framing is borrowed from Cura, which scaled curator preferences with a text encoder and reports leaving multimodal content untouched. That framing motivates the panel design and the homogenization concern, but we do not claim social effects.

Baselines

The naive reference point is plain recursive retraining with a fixed per-round budget and a single judge. The pluralistic-curation paper reports this collapses on CIFAR-10, with FID near 100. We run it on our data with identical generator, step budget, and sample counts, so that every other method's gain is measured against true collapse.

The direct competitor is pluralistic curation exactly as in Falahati et al., run with the same per-round budget as everything else. Its reported CIFAR-10 result (FID about 36 with five rewards versus about 100 with one) is the number we first try to reproduce in our pipeline before moving to audio. For the drum-loop setting, the reported anchor for the base generator is StyleGAN2 on Looperman with Inception Score 6.08 and Fréchet Audio Distance 2.22. We check that our round-zero model matches it.

The strongest fix suggested by the optimization literature is perturbed gradient descent from Jin et al.. It injects isotropic noise whenever the gradient norm falls below a threshold. We add it inside each round, with the noise radius and threshold tuned on the Gaussian-mixture grid under the same tuning budget given to our negative-curvature step. That step adds the power-iteration eigenvector found by the diagnostic, sign chosen to decrease loss, when $\lambda_{\min}$ is below a threshold.

The oracle-like baseline multiplies the per-round budget until results stop changing, approximating the exact fit the theory assumes. The key ablation keeps everything fixed and swaps entangled panels for separated panels. It isolates whether judge similarity, through curvature, is what drives the stall.

Beating the direct competitor means the following. At equal total compute, measured as gradient steps plus Hessian-vector products, the escape-aided runs end with per-judge reward spread and style-cluster coverage within noise of the large-budget oracle. The unaided pluralistic runs must stay measurably below it, and the gap must widen as panels become more entangled.

Why not just…?

Why not just train longer each round? This is the most serious threat. If the stall is purely a budget effect, the fix is trivial and our proposal reduces to a recommendation about step counts. The escape-time relation says the needed budget scales as $1/|\lambda_{\min}|$, so for entangled panels it may grow without useful bound. The large-budget oracle and the budget sweep will settle this directly. If a modest budget increase recovers the oracle at lower cost than any escape step, we will report that, and the diagnostic's remaining value is telling practitioners how many steps they need.

Why not just add noise, as perturbed gradient descent does? We expect this may well work. Our setting is plain likelihood training on curated data, not a min-max game. So the objection from Adolphs et al., that noise fails at dynamically stable false equilibria of games, does not obviously apply. Likewise the limit cycles that optimistic primal-dual methods were built to damp appear only if the bargaining weights are themselves learned. What could make noise insufficient is the degenerate curvature of deep generators, where isotropic noise spends most of its energy in flat directions. If perturbed gradient descent matches the negative-curvature step at equal compute, we recommend it and drop the curvature step. The diagnostic remains useful either way.

Why not just mix real data back in each round? Mixing real data is a common guard against collapse in self-consuming training. But it pulls the fixed point toward the original data distribution rather than toward the stakeholder compromise. It can therefore preserve diversity while ignoring what the judges asked for. We include it as a baseline and score it on both diversity and per-judge reward. It wins only if it matches the oracle on both.

Why not skip retraining and use best-of-N reranking? CMI-RewardBench shows that best-of-10 reranking with CMI-RM improves human preference. Reranking never enters the feedback loop, so it cannot collapse. It also cannot move the generator, and its cost scales with $N$ at every generation. We report it as a non-retraining reference on the same judges.

Proposed Experiments

The first decisive check is on Gaussian mixtures. Measured escape times across leakage gaps should track $1/(\eta|\lambda_{\min}|)$, and the finite-budget phase boundary should sit inside the infinite-capacity one. If escape time is insensitive to curvature, the mechanism is wrong and we stop there.

The second check is on CIFAR-10 at the published setting. Unaided pluralistic curation at reduced per-round budgets should lose the reported FID improvement while the oracle retains it. The diagnostic should flag the failing runs before diversity visibly drops.

The third check, the one that ties the directions together, is on Looperman drum loops. Over ten retraining rounds we track four measures:

  • Fréchet Audio Distance;
  • per-judge reward spread;
  • coverage of held-out style clusters;
  • a blinded listening test.

The listening test asks listeners to sort loops by style and to rate musicality on samples from rounds 0, 5, and 10. The predicted signature has four parts. Entangled panels stall while separated panels do not. Fréchet Audio Distance stays near its round-zero value even in stalled runs, while perceptual coverage and listener style-sorting accuracy fall. The escape-aided and large-budget runs restore coverage.

The proposal is falsified if stalled runs never appear at practical budgets. It is also falsified if they appear but are uncorrelated with the curvature diagnostic. A third falsifying outcome is that Fréchet Audio Distance detects the collapse as readily as the perceptual measures do, since the audio arm of the proposal rests on the claim that it cannot.